[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:24:32 dist-test-slave-ptl9 krb5kdc[12187](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:24:32 dist-test-slave-ptl9 krb5kdc[12187](info): set up 2 sockets
Sep 23 17:24:32 dist-test-slave-ptl9 krb5kdc[12187](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260923 17:24:33.905290 12180 mini_kdc.cc:121] Time spent starting KDC: real 2.267s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260923 17:24:34.161377 12180 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.256s user 0.002s sys 0.000s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260923 17:24:34.361024 12180 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.199s user 0.002s sys 0.000s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260923 17:24:34.618569 12180 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.257s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:24:34 dist-test-slave-ptl9 krb5kdc[12187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:24:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:24:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.229.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
I20260923 17:24:34.726656 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:44563
--webserver_interface=127.11.229.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:43815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:44563
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:34.839596 12200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:34.839910 12200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:34.839993 12200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:34.844065 12200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:24:34.844172 12200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:34.844213 12200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:34.844251 12200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:24:34.844285 12200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:24:34.849786 12200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:43815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:44563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:44563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:34.851074 12200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:34.852313 12200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 17:24:34.861970 12200 server_base.cc:1034] running on GCE node
W20260923 17:24:34.861984 12205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:34.861917 12206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:34.861919 12208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:34.862782 12200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:34.864078 12200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:34.865350 12200 hybrid_clock.cc:648] HybridClock initialized: now 1790184274865312 us; error 49 us; skew 500 ppm
Sep 23 17:24:34 dist-test-slave-ptl9 krb5kdc[12187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184274, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:34.868314 12200 init.cc:377] Logged in from keytab as kudu/127.11.229.62@KRBTEST.COM (short username kudu)
I20260923 17:24:34.869331 12200 webserver.cc:460] Webserver started at http://127.11.229.62:42227/ using document root <none> and password file <none>
I20260923 17:24:34.869942 12200 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:34.870005 12200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:34.870183 12200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:34.872460 12200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "91d84dbf6ad7491ca58c3dc8388759de"
format_stamp: "Formatted at 2026-09-23 17:24:34 on dist-test-slave-ptl9"
server_key: "7b8435cdfc4df671f43f2df1b0b2ff1a"
server_key_iv: "2eb40eb29ab2d904e659030f63433677"
server_key_version: "encryptionkey@0"
I20260923 17:24:34.873034 12200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "91d84dbf6ad7491ca58c3dc8388759de"
format_stamp: "Formatted at 2026-09-23 17:24:34 on dist-test-slave-ptl9"
server_key: "7b8435cdfc4df671f43f2df1b0b2ff1a"
server_key_iv: "2eb40eb29ab2d904e659030f63433677"
server_key_version: "encryptionkey@0"
I20260923 17:24:34.877817 12200 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:24:34.880517 12215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:34.882017 12200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:24:34.882167 12200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "91d84dbf6ad7491ca58c3dc8388759de"
format_stamp: "Formatted at 2026-09-23 17:24:34 on dist-test-slave-ptl9"
server_key: "7b8435cdfc4df671f43f2df1b0b2ff1a"
server_key_iv: "2eb40eb29ab2d904e659030f63433677"
server_key_version: "encryptionkey@0"
I20260923 17:24:34.882284 12200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:34.893473 12200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:34.897564 12200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:34.897784 12200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:34.905604 12200 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:44563
I20260923 17:24:34.905627 12267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:44563 every 8 connection(s)
I20260923 17:24:34.906710 12200 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:24:34.910023 12268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:34.913604 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 12200
I20260923 17:24:34.913789 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:24:34.915128 12180 external_mini_cluster.cc:1428] Setting key 51ae1fe7d667dc5bde1507db9a98d530
I20260923 17:24:34.916448 12268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91d84dbf6ad7491ca58c3dc8388759de: Bootstrap starting.
I20260923 17:24:34.919322 12268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91d84dbf6ad7491ca58c3dc8388759de: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:34.920346 12268 log.cc:824] T 00000000000000000000000000000000 P 91d84dbf6ad7491ca58c3dc8388759de: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:34.922967 12268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91d84dbf6ad7491ca58c3dc8388759de: No bootstrap required, opened a new log
I20260923 17:24:34.928460 12268 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91d84dbf6ad7491ca58c3dc8388759de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91d84dbf6ad7491ca58c3dc8388759de" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 44563 } }
I20260923 17:24:34.928817 12268 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91d84dbf6ad7491ca58c3dc8388759de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:34.928913 12268 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91d84dbf6ad7491ca58c3dc8388759de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d84dbf6ad7491ca58c3dc8388759de, State: Initialized, Role: FOLLOWER
I20260923 17:24:34.929528 12268 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91d84dbf6ad7491ca58c3dc8388759de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91d84dbf6ad7491ca58c3dc8388759de" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 44563 } }
I20260923 17:24:34.929667 12268 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91d84dbf6ad7491ca58c3dc8388759de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:24:34.929761 12268 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91d84dbf6ad7491ca58c3dc8388759de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:24:34.929889 12268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91d84dbf6ad7491ca58c3dc8388759de [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:34.931221 12268 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91d84dbf6ad7491ca58c3dc8388759de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91d84dbf6ad7491ca58c3dc8388759de" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 44563 } }
I20260923 17:24:34.931628 12268 leader_election.cc:302] T 00000000000000000000000000000000 P 91d84dbf6ad7491ca58c3dc8388759de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91d84dbf6ad7491ca58c3dc8388759de; no voters:
I20260923 17:24:34.932051 12268 leader_election.cc:288] T 00000000000000000000000000000000 P 91d84dbf6ad7491ca58c3dc8388759de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:24:34.932173 12273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91d84dbf6ad7491ca58c3dc8388759de [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:24:34.932472 12273 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91d84dbf6ad7491ca58c3dc8388759de [term 1 LEADER]: Becoming Leader. State: Replica: 91d84dbf6ad7491ca58c3dc8388759de, State: Running, Role: LEADER
I20260923 17:24:34.932873 12273 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91d84dbf6ad7491ca58c3dc8388759de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91d84dbf6ad7491ca58c3dc8388759de" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 44563 } }
I20260923 17:24:34.933598 12268 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91d84dbf6ad7491ca58c3dc8388759de [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:24:34.934772 12274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91d84dbf6ad7491ca58c3dc8388759de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91d84dbf6ad7491ca58c3dc8388759de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91d84dbf6ad7491ca58c3dc8388759de" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 44563 } } }
I20260923 17:24:34.934899 12274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91d84dbf6ad7491ca58c3dc8388759de [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:34.935398 12275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91d84dbf6ad7491ca58c3dc8388759de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91d84dbf6ad7491ca58c3dc8388759de. Latest consensus state: current_term: 1 leader_uuid: "91d84dbf6ad7491ca58c3dc8388759de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91d84dbf6ad7491ca58c3dc8388759de" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 44563 } } }
I20260923 17:24:34.935570 12275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91d84dbf6ad7491ca58c3dc8388759de [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:34.935886 12282 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 23 17:24:34 dist-test-slave-ptl9 krb5kdc[12187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:24:34.939899 12282 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:24:34.945391 12270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:34.918618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35324 (local address 127.11.229.62:44563)
0923 17:24:34.919182 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:34.919204 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:34.920290 (+ 1086us) server_negotiation.cc:407] Connection header received
0923 17:24:34.924566 (+ 4276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:34.924586 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:34.924948 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:34.925253 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:34.929005 (+ 3752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:34.930409 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:34.932311 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:34.933115 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:34.933318 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:34.938826 (+ 5508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:34.938832 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:34.938848 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:34.938880 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:34.940974 (+ 2094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:34.941695 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:34.941702 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:34.941710 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:34.941837 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:34.942316 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:34.942320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:34.942322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:34.942943 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:34.943198 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:34.943947 (+ 749us) server_negotiation.cc:299] Negotiation successful
0923 17:24:34.944289 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":159,"threads_started":1}
I20260923 17:24:34.946923 12282 catalog_manager.cc:1347] Generated new cluster ID: cedb513a81d84989bb6f2ce6cc719284
I20260923 17:24:34.947072 12282 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:24:34.971232 12282 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:24:34.972676 12282 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:24:34.983222 12282 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91d84dbf6ad7491ca58c3dc8388759de: Generated new TSK 0
I20260923 17:24:34.984213 12282 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.229.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab.
Entry for principal kudu/127.11.229.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab.
I20260923 17:24:35.017170 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.1:0
--local_ip_for_outbound_sockets=127.11.229.1
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:44563
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:43815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:35.136468 12294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:35.136742 12294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:35.136821 12294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:35.141079 12294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:35.141139 12294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:35.141278 12294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.1
I20260923 17:24:35.146682 12294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:43815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:44563
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.1
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:35.147758 12294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:35.148810 12294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:35.159217 12300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:35.159202 12299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:35.159189 12302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:35.159441 12294 server_base.cc:1034] running on GCE node
I20260923 17:24:35.159929 12294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:35.160604 12294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:35.161769 12294 hybrid_clock.cc:648] HybridClock initialized: now 1790184275161732 us; error 53 us; skew 500 ppm
Sep 23 17:24:35 dist-test-slave-ptl9 krb5kdc[12187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184275, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:35.164563 12294 init.cc:377] Logged in from keytab as kudu/127.11.229.1@KRBTEST.COM (short username kudu)
I20260923 17:24:35.165566 12294 webserver.cc:460] Webserver started at http://127.11.229.1:43195/ using document root <none> and password file <none>
I20260923 17:24:35.166158 12294 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:35.166252 12294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:35.166483 12294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:35.168571 12294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-0/data/instance:
uuid: "40974f00e69b422581405c9891d5af35"
format_stamp: "Formatted at 2026-09-23 17:24:35 on dist-test-slave-ptl9"
server_key: "9953afda3e4b7f83ea933a41c88af3a7"
server_key_iv: "825ec46fd6de150e489c778af4726840"
server_key_version: "encryptionkey@0"
I20260923 17:24:35.169220 12294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance:
uuid: "40974f00e69b422581405c9891d5af35"
format_stamp: "Formatted at 2026-09-23 17:24:35 on dist-test-slave-ptl9"
server_key: "9953afda3e4b7f83ea933a41c88af3a7"
server_key_iv: "825ec46fd6de150e489c778af4726840"
server_key_version: "encryptionkey@0"
I20260923 17:24:35.173069 12294 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:24:35.175638 12309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:35.176954 12294 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:24:35.177114 12294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-0/wal
uuid: "40974f00e69b422581405c9891d5af35"
format_stamp: "Formatted at 2026-09-23 17:24:35 on dist-test-slave-ptl9"
server_key: "9953afda3e4b7f83ea933a41c88af3a7"
server_key_iv: "825ec46fd6de150e489c778af4726840"
server_key_version: "encryptionkey@0"
I20260923 17:24:35.177240 12294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:35.192142 12294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:35.195732 12294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:35.195968 12294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:35.196638 12294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:24:35.197575 12294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:24:35.197650 12294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:35.197724 12294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:24:35.197777 12294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:35.207691 12294 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.1:38675
I20260923 17:24:35.207713 12422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.1:38675 every 8 connection(s)
I20260923 17:24:35.208684 12294 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
I20260923 17:24:35.214507 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 12294
I20260923 17:24:35.214623 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance
I20260923 17:24:35.214890 12180 external_mini_cluster.cc:1428] Setting key b37985f0146155a9c0b9106be2a0d98d
Sep 23 17:24:35 dist-test-slave-ptl9 krb5kdc[12187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184275, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:24:35.224265 12270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:35.210768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:59523 (local address 127.11.229.62:44563)
0923 17:24:35.210919 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:35.210923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:35.211980 (+ 1057us) server_negotiation.cc:407] Connection header received
0923 17:24:35.212904 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:35.212907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:35.212999 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:35.213085 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:35.214436 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:35.215292 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:35.216896 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:35.217358 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:35.217482 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:35.220182 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:35.220185 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:35.220188 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:35.220213 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:35.221820 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:35.222519 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:35.222523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:35.222524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:35.222607 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:35.223021 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:35.223023 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:35.223024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:35.223214 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:35.223290 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:35.223991 (+ 701us) server_negotiation.cc:299] Negotiation successful
0923 17:24:35.224115 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
WARNING: no policy specified for kudu/127.11.229.2@KRBTEST.COM; defaulting to no policy
I20260923 17:24:35.225584 12425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:35.211111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:44563 (local address 127.11.229.1:59523)
0923 17:24:35.211806 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:35.211850 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:35.212651 (+ 801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:35.213347 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:35.213355 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:35.213745 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:35.214254 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:35.214269 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:35.215446 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:35.215449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:35.216759 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:35.216770 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:35.217509 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:35.217512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:35.217665 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:35.218353 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:35.218370 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:35.219995 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:35.221947 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:35.221956 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:35.221989 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:35.222402 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:35.222704 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:35.222709 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:35.222711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:35.222915 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:35.223329 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:35.223337 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:35.223713 (+ 376us) client_negotiation.cc:769] Sending connection context
0923 17:24:35.223956 (+ 243us) client_negotiation.cc:240] Negotiation successful
0923 17:24:35.224354 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":116,"threads_started":1}
I20260923 17:24:35.226979 12423 heartbeater.cc:347] Connected to a master server at 127.11.229.62:44563
I20260923 17:24:35.227358 12423 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.2@KRBTEST.COM" created.
I20260923 17:24:35.228016 12423 heartbeater.cc:511] Master 127.11.229.62:44563 requested a full tablet report, sending...
I20260923 17:24:35.229854 12232 ts_manager.cc:194] Registered new tserver with Master: 40974f00e69b422581405c9891d5af35 (127.11.229.1:38675)
I20260923 17:24:35.231004 12232 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.1@KRBTEST.COM'} at 127.11.229.1:59523
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab.
Entry for principal kudu/127.11.229.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab.
I20260923 17:24:35.244349 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.2:0
--local_ip_for_outbound_sockets=127.11.229.2
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:44563
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:43815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:35.358846 12428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:35.359117 12428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:35.359163 12428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:35.363317 12428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:35.363375 12428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:35.363435 12428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.2
I20260923 17:24:35.368641 12428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:43815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:44563
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.2
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:35.369777 12428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:35.370836 12428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:35.380748 12436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:35.380735 12433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:35.380915 12428 server_base.cc:1034] running on GCE node
W20260923 17:24:35.380735 12434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:35.381596 12428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:35.382321 12428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:35.383522 12428 hybrid_clock.cc:648] HybridClock initialized: now 1790184275383503 us; error 50 us; skew 500 ppm
Sep 23 17:24:35 dist-test-slave-ptl9 krb5kdc[12187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184275, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:35.386534 12428 init.cc:377] Logged in from keytab as kudu/127.11.229.2@KRBTEST.COM (short username kudu)
I20260923 17:24:35.387533 12428 webserver.cc:460] Webserver started at http://127.11.229.2:37327/ using document root <none> and password file <none>
I20260923 17:24:35.388195 12428 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:35.388262 12428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:35.388515 12428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:35.390815 12428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-1/data/instance:
uuid: "ca76094969754611ad47f4eeac01c04f"
format_stamp: "Formatted at 2026-09-23 17:24:35 on dist-test-slave-ptl9"
server_key: "01836ab8374f9c5c4d5e73b08feab045"
server_key_iv: "eec64d3720c1e01250552be39bd292d4"
server_key_version: "encryptionkey@0"
I20260923 17:24:35.391438 12428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance:
uuid: "ca76094969754611ad47f4eeac01c04f"
format_stamp: "Formatted at 2026-09-23 17:24:35 on dist-test-slave-ptl9"
server_key: "01836ab8374f9c5c4d5e73b08feab045"
server_key_iv: "eec64d3720c1e01250552be39bd292d4"
server_key_version: "encryptionkey@0"
I20260923 17:24:35.395735 12428 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260923 17:24:35.398387 12443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:35.399703 12428 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:24:35.399833 12428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-1/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-1/wal
uuid: "ca76094969754611ad47f4eeac01c04f"
format_stamp: "Formatted at 2026-09-23 17:24:35 on dist-test-slave-ptl9"
server_key: "01836ab8374f9c5c4d5e73b08feab045"
server_key_iv: "eec64d3720c1e01250552be39bd292d4"
server_key_version: "encryptionkey@0"
I20260923 17:24:35.399951 12428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:35.418615 12428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:35.422120 12428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:35.422333 12428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:35.422950 12428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:24:35.423806 12428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:24:35.423852 12428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:35.423920 12428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:24:35.423951 12428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:35.433501 12428 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.2:42101
I20260923 17:24:35.433501 12556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.2:42101 every 8 connection(s)
I20260923 17:24:35.434490 12428 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
I20260923 17:24:35.441322 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 12428
I20260923 17:24:35.441430 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance
I20260923 17:24:35.441721 12180 external_mini_cluster.cc:1428] Setting key 2ba940921d65b6766774599aa5c09a6f
Sep 23 17:24:35 dist-test-slave-ptl9 krb5kdc[12187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184275, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:24:35.449846 12270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:35.436413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:35415 (local address 127.11.229.62:44563)
0923 17:24:35.436548 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:35.436551 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:35.437747 (+ 1196us) server_negotiation.cc:407] Connection header received
0923 17:24:35.438674 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:35.438678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:35.438738 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:35.438819 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:35.440024 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:35.440572 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:35.442066 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:35.442317 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:35.442438 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:35.445455 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:35.445458 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:35.445460 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:35.445487 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:35.447368 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:35.448058 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:35.448061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:35.448073 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:35.448153 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:35.448582 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:35.448584 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:35.448585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:35.448769 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:35.448843 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:35.449609 (+ 766us) server_negotiation.cc:299] Negotiation successful
0923 17:24:35.449717 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 17:24:35.451179 12559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:35.436867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:44563 (local address 127.11.229.2:35415)
0923 17:24:35.437576 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:35.437617 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:35.438457 (+ 840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:35.438993 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:35.439000 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:35.439373 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:35.439872 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:35.439887 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:35.440707 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:35.440716 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:35.441937 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:35.441948 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:35.442482 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:35.442486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:35.442705 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:35.443352 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:35.443368 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:35.445181 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:35.447519 (+ 2338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:35.447528 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:35.447551 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:35.447938 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:35.448255 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:35.448259 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:35.448263 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:35.448475 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:35.448881 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:35.448890 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:35.449314 (+ 424us) client_negotiation.cc:769] Sending connection context
0923 17:24:35.449560 (+ 246us) client_negotiation.cc:240] Negotiation successful
0923 17:24:35.449957 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for kudu/127.11.229.3@KRBTEST.COM; defaulting to no policy
I20260923 17:24:35.452548 12557 heartbeater.cc:347] Connected to a master server at 127.11.229.62:44563
I20260923 17:24:35.452816 12557 heartbeater.cc:464] Registering TS with master...
I20260923 17:24:35.453429 12557 heartbeater.cc:511] Master 127.11.229.62:44563 requested a full tablet report, sending...
I20260923 17:24:35.454628 12232 ts_manager.cc:194] Registered new tserver with Master: ca76094969754611ad47f4eeac01c04f (127.11.229.2:42101)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.3@KRBTEST.COM" created.
I20260923 17:24:35.455417 12232 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.2@KRBTEST.COM'} at 127.11.229.2:35415
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab.
Entry for principal kudu/127.11.229.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab.
I20260923 17:24:35.474653 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.3:0
--local_ip_for_outbound_sockets=127.11.229.3
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:44563
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:43815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:35.590642 12562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:35.590950 12562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:35.591058 12562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:35.595119 12562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:35.595201 12562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:35.595286 12562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.3
I20260923 17:24:35.600534 12562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:43815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:44563
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.3
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:35.601634 12562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:35.602661 12562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:35.612562 12568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:35.612541 12567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:35.612635 12562 server_base.cc:1034] running on GCE node
W20260923 17:24:35.612566 12570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:35.613325 12562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:35.614029 12562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:35.615253 12562 hybrid_clock.cc:648] HybridClock initialized: now 1790184275615229 us; error 35 us; skew 500 ppm
Sep 23 17:24:35 dist-test-slave-ptl9 krb5kdc[12187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184275, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:35.618283 12562 init.cc:377] Logged in from keytab as kudu/127.11.229.3@KRBTEST.COM (short username kudu)
I20260923 17:24:35.619230 12562 webserver.cc:460] Webserver started at http://127.11.229.3:44271/ using document root <none> and password file <none>
I20260923 17:24:35.619838 12562 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:35.619935 12562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:35.620178 12562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:35.622444 12562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-2/data/instance:
uuid: "b371dfa8ad294c2c8f443ee7a5c568c2"
format_stamp: "Formatted at 2026-09-23 17:24:35 on dist-test-slave-ptl9"
server_key: "cbceb8213f9319cf26a25313509f59e0"
server_key_iv: "5b9e7c92bce19bb8f7d6b7be14e79c39"
server_key_version: "encryptionkey@0"
I20260923 17:24:35.623039 12562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance:
uuid: "b371dfa8ad294c2c8f443ee7a5c568c2"
format_stamp: "Formatted at 2026-09-23 17:24:35 on dist-test-slave-ptl9"
server_key: "cbceb8213f9319cf26a25313509f59e0"
server_key_iv: "5b9e7c92bce19bb8f7d6b7be14e79c39"
server_key_version: "encryptionkey@0"
I20260923 17:24:35.627246 12562 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:24:35.629731 12577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:35.631018 12562 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:24:35.631173 12562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-2/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-2/wal
uuid: "b371dfa8ad294c2c8f443ee7a5c568c2"
format_stamp: "Formatted at 2026-09-23 17:24:35 on dist-test-slave-ptl9"
server_key: "cbceb8213f9319cf26a25313509f59e0"
server_key_iv: "5b9e7c92bce19bb8f7d6b7be14e79c39"
server_key_version: "encryptionkey@0"
I20260923 17:24:35.631294 12562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:35.653781 12562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:35.656842 12562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:35.657059 12562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:35.657708 12562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:24:35.658558 12562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:24:35.658638 12562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:35.658730 12562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:24:35.658758 12562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:35.668478 12562 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.3:43037
I20260923 17:24:35.668499 12690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.3:43037 every 8 connection(s)
I20260923 17:24:35.669562 12562 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
I20260923 17:24:35.671697 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 12562
I20260923 17:24:35.671818 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance
I20260923 17:24:35.672108 12180 external_mini_cluster.cc:1428] Setting key e1e4920b15b933e50c8879397ab573ca
Sep 23 17:24:35 dist-test-slave-ptl9 krb5kdc[12187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184275, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:24:35.683593 12270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:35.671520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:36769 (local address 127.11.229.62:44563)
0923 17:24:35.671666 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:35.671677 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:35.673319 (+ 1642us) server_negotiation.cc:407] Connection header received
0923 17:24:35.674269 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:35.674273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:35.674331 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:35.674440 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:35.675644 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:35.676195 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:35.677379 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:35.677606 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:35.677723 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:35.680196 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:35.680199 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:35.680201 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:35.680225 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:35.681775 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:35.682237 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:35.682246 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:35.682247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:35.682325 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:35.682623 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:35.682626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:35.682627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:35.682805 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:35.682881 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:35.683347 (+ 466us) server_negotiation.cc:299] Negotiation successful
0923 17:24:35.683449 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 17:24:35.684393 12693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:35.671974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:44563 (local address 127.11.229.3:36769)
0923 17:24:35.673147 (+ 1173us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:35.673190 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:35.674063 (+ 873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:35.674626 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:35.674633 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:35.675009 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:35.675497 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:35.675510 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:35.676321 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:35.676325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:35.677213 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:35.677222 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:35.677760 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:35.677764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:35.677937 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:35.678540 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:35.678557 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:35.679999 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:35.681876 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:35.681881 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:35.681893 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:35.682145 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:35.682404 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:35.682407 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:35.682408 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:35.682540 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:35.682893 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:35.682898 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:35.683131 (+ 233us) client_negotiation.cc:769] Sending connection context
0923 17:24:35.683287 (+ 156us) client_negotiation.cc:240] Negotiation successful
0923 17:24:35.683539 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":948,"thread_start_us":158,"threads_started":1}
I20260923 17:24:35.685729 12691 heartbeater.cc:347] Connected to a master server at 127.11.229.62:44563
I20260923 17:24:35.685995 12691 heartbeater.cc:464] Registering TS with master...
I20260923 17:24:35.686573 12691 heartbeater.cc:511] Master 127.11.229.62:44563 requested a full tablet report, sending...
I20260923 17:24:35.687590 12232 ts_manager.cc:194] Registered new tserver with Master: b371dfa8ad294c2c8f443ee7a5c568c2 (127.11.229.3:43037)
I20260923 17:24:35.688172 12232 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.3@KRBTEST.COM'} at 127.11.229.3:36769
I20260923 17:24:35.698751 12180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 17:24:35 dist-test-slave-ptl9 krb5kdc[12187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184275, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 17:24:35 dist-test-slave-ptl9 krb5kdc[12187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184275, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.229.1@KRBTEST.COM
I20260923 17:24:35.725345 12698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:35.716023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34264 (local address 127.11.229.1:38675)
0923 17:24:35.716351 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:35.716365 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:35.716392 (+ 27us) server_negotiation.cc:407] Connection header received
0923 17:24:35.716452 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:35.716457 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:35.716666 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:35.716792 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:35.717506 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:35.718338 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:35.719135 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:35.719506 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:35.719585 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:35.722135 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:35.722145 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:35.722160 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:35.722186 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:35.723639 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:35.724073 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:35.724079 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:35.724080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:35.724168 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:35.724400 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:35.724403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:35.724404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:35.724641 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:35.724796 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:35.724946 (+ 150us) server_negotiation.cc:299] Negotiation successful
0923 17:24:35.725108 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":110,"threads_started":1}
W20260923 17:24:35.726425 12337 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34264
Sep 23 17:24:35 dist-test-slave-ptl9 krb5kdc[12187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 17:24:35 dist-test-slave-ptl9 krb5kdc[12187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.1@KRBTEST.COM
I20260923 17:24:35.752290 12698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:35.744007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34272 (local address 127.11.229.1:38675)
0923 17:24:35.744173 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:35.744178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:35.744317 (+ 139us) server_negotiation.cc:407] Connection header received
0923 17:24:35.744503 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:35.744506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:35.744552 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:35.744636 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:35.745370 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:35.745912 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:35.746745 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:35.746988 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:35.747063 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:35.749412 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:35.749415 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:35.749417 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:35.749442 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:35.750670 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:35.751080 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:35.751084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:35.751086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:35.751165 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:35.751439 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:35.751443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:35.751445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:35.751619 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:35.751755 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:35.751993 (+ 238us) server_negotiation.cc:299] Negotiation successful
0923 17:24:35.752124 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 17:24:35.753397 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 12294
I20260923 17:24:35.759793 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 12428
I20260923 17:24:35.767558 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 12562
I20260923 17:24:35.773773 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 12200
2026-09-23T17:24:35Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4168 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:24:35 dist-test-slave-ptl9 krb5kdc[12705](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:24:35 dist-test-slave-ptl9 krb5kdc[12705](info): set up 2 sockets
Sep 23 17:24:35 dist-test-slave-ptl9 krb5kdc[12705](info): commencing operation
krb5kdc: starting...
W20260923 17:24:36.809192 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:24:36 dist-test-slave-ptl9 krb5kdc[12705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:24:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:24:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.229.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
I20260923 17:24:36.932637 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:45773
--webserver_interface=127.11.229.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:38607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:45773
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:37.044405 12718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:37.044713 12718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:37.044795 12718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:37.048899 12718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:24:37.049000 12718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:37.049048 12718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:37.049103 12718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:24:37.049141 12718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:24:37.054493 12718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:38607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:45773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:45773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:37.055583 12718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:37.056641 12718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:37.065115 12723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:37.065125 12724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:37.065124 12726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:37.065173 12718 server_base.cc:1034] running on GCE node
I20260923 17:24:37.065929 12718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:37.067054 12718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:37.068234 12718 hybrid_clock.cc:648] HybridClock initialized: now 1790184277068213 us; error 33 us; skew 500 ppm
Sep 23 17:24:37 dist-test-slave-ptl9 krb5kdc[12705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184277, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:37.074159 12718 init.cc:377] Logged in from keytab as kudu/127.11.229.62@KRBTEST.COM (short username kudu)
I20260923 17:24:37.075158 12718 webserver.cc:460] Webserver started at http://127.11.229.62:33581/ using document root <none> and password file <none>
I20260923 17:24:37.075745 12718 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:37.075837 12718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:37.076083 12718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:37.078292 12718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "60c36c0047a548d69e1241ddfa90459b"
format_stamp: "Formatted at 2026-09-23 17:24:37 on dist-test-slave-ptl9"
server_key: "a62dc7975304218c573f57280753ce8d"
server_key_iv: "098b739d00eacaa1839d2b3a5e216524"
server_key_version: "encryptionkey@0"
I20260923 17:24:37.078867 12718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "60c36c0047a548d69e1241ddfa90459b"
format_stamp: "Formatted at 2026-09-23 17:24:37 on dist-test-slave-ptl9"
server_key: "a62dc7975304218c573f57280753ce8d"
server_key_iv: "098b739d00eacaa1839d2b3a5e216524"
server_key_version: "encryptionkey@0"
I20260923 17:24:37.082654 12718 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:24:37.085012 12733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:37.086378 12718 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 17:24:37.086571 12718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "60c36c0047a548d69e1241ddfa90459b"
format_stamp: "Formatted at 2026-09-23 17:24:37 on dist-test-slave-ptl9"
server_key: "a62dc7975304218c573f57280753ce8d"
server_key_iv: "098b739d00eacaa1839d2b3a5e216524"
server_key_version: "encryptionkey@0"
I20260923 17:24:37.086700 12718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:37.110862 12718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:37.114037 12718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:37.114245 12718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:37.121608 12718 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:45773
I20260923 17:24:37.121626 12785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:45773 every 8 connection(s)
I20260923 17:24:37.122601 12718 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:24:37.125375 12786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:37.129228 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 12718
I20260923 17:24:37.129401 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:24:37.129655 12180 external_mini_cluster.cc:1428] Setting key 8c07edbd792e0ba67d157d022d79e4a7
I20260923 17:24:37.131181 12786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60c36c0047a548d69e1241ddfa90459b: Bootstrap starting.
I20260923 17:24:37.133491 12786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60c36c0047a548d69e1241ddfa90459b: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:37.134246 12786 log.cc:824] T 00000000000000000000000000000000 P 60c36c0047a548d69e1241ddfa90459b: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:37.136052 12786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60c36c0047a548d69e1241ddfa90459b: No bootstrap required, opened a new log
Sep 23 17:24:37 dist-test-slave-ptl9 krb5kdc[12705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:24:37.139999 12786 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60c36c0047a548d69e1241ddfa90459b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60c36c0047a548d69e1241ddfa90459b" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 45773 } }
I20260923 17:24:37.140250 12786 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60c36c0047a548d69e1241ddfa90459b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:37.140338 12786 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60c36c0047a548d69e1241ddfa90459b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60c36c0047a548d69e1241ddfa90459b, State: Initialized, Role: FOLLOWER
I20260923 17:24:37.140849 12786 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60c36c0047a548d69e1241ddfa90459b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60c36c0047a548d69e1241ddfa90459b" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 45773 } }
I20260923 17:24:37.141012 12786 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60c36c0047a548d69e1241ddfa90459b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:24:37.141088 12786 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60c36c0047a548d69e1241ddfa90459b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:24:37.141201 12786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60c36c0047a548d69e1241ddfa90459b [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:37.142328 12786 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60c36c0047a548d69e1241ddfa90459b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60c36c0047a548d69e1241ddfa90459b" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 45773 } }
I20260923 17:24:37.142676 12786 leader_election.cc:302] T 00000000000000000000000000000000 P 60c36c0047a548d69e1241ddfa90459b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60c36c0047a548d69e1241ddfa90459b; no voters:
I20260923 17:24:37.143026 12786 leader_election.cc:288] T 00000000000000000000000000000000 P 60c36c0047a548d69e1241ddfa90459b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:24:37.143160 12791 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60c36c0047a548d69e1241ddfa90459b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:24:37.143468 12791 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60c36c0047a548d69e1241ddfa90459b [term 1 LEADER]: Becoming Leader. State: Replica: 60c36c0047a548d69e1241ddfa90459b, State: Running, Role: LEADER
I20260923 17:24:37.143811 12791 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60c36c0047a548d69e1241ddfa90459b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60c36c0047a548d69e1241ddfa90459b" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 45773 } }
I20260923 17:24:37.144346 12786 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60c36c0047a548d69e1241ddfa90459b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:24:37.144586 12789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:37.131145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35532 (local address 127.11.229.62:45773)
0923 17:24:37.131583 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:37.131598 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:37.131634 (+ 36us) server_negotiation.cc:407] Connection header received
0923 17:24:37.132260 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:37.132276 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:37.132610 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:37.132960 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:37.133799 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:37.134852 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:37.135698 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:37.136200 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:37.136381 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:37.139236 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:37.139241 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:37.139254 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:37.139280 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:37.141341 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:37.141778 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:37.141784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:37.141791 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:37.141930 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:37.142208 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:37.142214 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:37.142218 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:37.142745 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:37.142936 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:37.143350 (+ 414us) server_negotiation.cc:299] Negotiation successful
0923 17:24:37.143651 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":130,"threads_started":1}
I20260923 17:24:37.145005 12791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60c36c0047a548d69e1241ddfa90459b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60c36c0047a548d69e1241ddfa90459b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60c36c0047a548d69e1241ddfa90459b" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 45773 } } }
I20260923 17:24:37.145133 12791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60c36c0047a548d69e1241ddfa90459b [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:37.145399 12791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60c36c0047a548d69e1241ddfa90459b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60c36c0047a548d69e1241ddfa90459b. Latest consensus state: current_term: 1 leader_uuid: "60c36c0047a548d69e1241ddfa90459b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60c36c0047a548d69e1241ddfa90459b" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 45773 } } }
I20260923 17:24:37.145573 12791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60c36c0047a548d69e1241ddfa90459b [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:37.145924 12798 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:24:37.148867 12798 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:24:37.154501 12798 catalog_manager.cc:1347] Generated new cluster ID: cb15b900df7e429e9d51a99bd549bd56
I20260923 17:24:37.154588 12798 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:24:37.171407 12798 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:24:37.172973 12798 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:24:37.182166 12798 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60c36c0047a548d69e1241ddfa90459b: Generated new TSK 0
I20260923 17:24:37.183230 12798 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.229.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab.
Entry for principal kudu/127.11.229.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab.
I20260923 17:24:37.224367 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.1:0
--local_ip_for_outbound_sockets=127.11.229.1
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:45773
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:38607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:37.363163 12812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:37.363428 12812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:37.363473 12812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:37.367427 12812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:37.367481 12812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:37.367540 12812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.1
I20260923 17:24:37.372656 12812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:38607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:45773
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.1
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:37.373744 12812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:37.374751 12812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:37.384563 12818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:37.384581 12817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:37.384559 12820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:37.384702 12812 server_base.cc:1034] running on GCE node
I20260923 17:24:37.385335 12812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:37.385991 12812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:37.387162 12812 hybrid_clock.cc:648] HybridClock initialized: now 1790184277387146 us; error 28 us; skew 500 ppm
Sep 23 17:24:37 dist-test-slave-ptl9 krb5kdc[12705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184277, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:37.398573 12812 init.cc:377] Logged in from keytab as kudu/127.11.229.1@KRBTEST.COM (short username kudu)
I20260923 17:24:37.399647 12812 webserver.cc:460] Webserver started at http://127.11.229.1:38775/ using document root <none> and password file <none>
I20260923 17:24:37.400273 12812 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:37.400368 12812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:37.400617 12812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:37.402855 12812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-0/data/instance:
uuid: "86c5cc4be68e46e5977830c35abed64a"
format_stamp: "Formatted at 2026-09-23 17:24:37 on dist-test-slave-ptl9"
server_key: "e29fb291b5ad9b1db50d36cc2243e31b"
server_key_iv: "402ac86e0a9476b5cc214a45ae7da077"
server_key_version: "encryptionkey@0"
I20260923 17:24:37.403451 12812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance:
uuid: "86c5cc4be68e46e5977830c35abed64a"
format_stamp: "Formatted at 2026-09-23 17:24:37 on dist-test-slave-ptl9"
server_key: "e29fb291b5ad9b1db50d36cc2243e31b"
server_key_iv: "402ac86e0a9476b5cc214a45ae7da077"
server_key_version: "encryptionkey@0"
I20260923 17:24:37.407382 12812 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:24:37.409957 12827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:37.411227 12812 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:24:37.411376 12812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-0/wal
uuid: "86c5cc4be68e46e5977830c35abed64a"
format_stamp: "Formatted at 2026-09-23 17:24:37 on dist-test-slave-ptl9"
server_key: "e29fb291b5ad9b1db50d36cc2243e31b"
server_key_iv: "402ac86e0a9476b5cc214a45ae7da077"
server_key_version: "encryptionkey@0"
I20260923 17:24:37.411499 12812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:37.440665 12812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:37.444486 12812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:37.444754 12812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:37.445506 12812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:24:37.446623 12812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:24:37.446708 12812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:37.446784 12812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:24:37.446826 12812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:37.456303 12812 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.1:36031
I20260923 17:24:37.456332 12940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.1:36031 every 8 connection(s)
I20260923 17:24:37.457311 12812 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
I20260923 17:24:37.461972 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 12812
I20260923 17:24:37.462057 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance
I20260923 17:24:37.462307 12180 external_mini_cluster.cc:1428] Setting key c8b598bb9f87b1379f271ce60869c931
Sep 23 17:24:37 dist-test-slave-ptl9 krb5kdc[12705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184277, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.229.2@KRBTEST.COM; defaulting to no policy
I20260923 17:24:37.472823 12789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:37.459312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:40013 (local address 127.11.229.62:45773)
0923 17:24:37.459473 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:37.459477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:37.460537 (+ 1060us) server_negotiation.cc:407] Connection header received
0923 17:24:37.461558 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:37.461561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:37.461618 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:37.461694 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:37.463187 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:37.463927 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:37.465757 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:37.466186 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:37.466307 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:37.469122 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:37.469126 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:37.469128 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:37.469152 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:37.470639 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:37.471176 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:37.471180 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:37.471182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:37.471264 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:37.471648 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:37.471652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:37.471654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:37.471844 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:37.471956 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:37.472519 (+ 563us) server_negotiation.cc:299] Negotiation successful
0923 17:24:37.472653 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 17:24:37.473876 12943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:37.459702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:45773 (local address 127.11.229.1:40013)
0923 17:24:37.460402 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:37.460446 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:37.461359 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:37.461933 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:37.461939 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:37.462435 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:37.463027 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:37.463057 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:37.464062 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:37.464067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:37.465616 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:37.465632 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:37.466353 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:37.466356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:37.466520 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:37.467166 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:37.467191 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:37.468930 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:37.470781 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:37.470787 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:37.470804 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:37.471069 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:37.471362 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:37.471365 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:37.471367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:37.471534 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:37.471972 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:37.471980 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:37.472279 (+ 299us) client_negotiation.cc:769] Sending connection context
0923 17:24:37.472479 (+ 200us) client_negotiation.cc:240] Negotiation successful
0923 17:24:37.472831 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":118,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.2@KRBTEST.COM" created.
I20260923 17:24:37.475361 12941 heartbeater.cc:347] Connected to a master server at 127.11.229.62:45773
I20260923 17:24:37.475696 12941 heartbeater.cc:464] Registering TS with master...
I20260923 17:24:37.476298 12941 heartbeater.cc:511] Master 127.11.229.62:45773 requested a full tablet report, sending...
I20260923 17:24:37.478111 12750 ts_manager.cc:194] Registered new tserver with Master: 86c5cc4be68e46e5977830c35abed64a (127.11.229.1:36031)
I20260923 17:24:37.479849 12750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.1@KRBTEST.COM'} at 127.11.229.1:40013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab.
Entry for principal kudu/127.11.229.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab.
I20260923 17:24:37.493489 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.2:0
--local_ip_for_outbound_sockets=127.11.229.2
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:45773
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:38607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:37.607916 12946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:37.608212 12946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:37.608258 12946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:37.612483 12946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:37.612556 12946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:37.612648 12946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.2
I20260923 17:24:37.618558 12946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:38607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:45773
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.2
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:37.619776 12946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:37.620865 12946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:37.631618 12952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:37.631697 12951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:37.631706 12954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:37.631779 12946 server_base.cc:1034] running on GCE node
I20260923 17:24:37.632268 12946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:37.633000 12946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:37.634212 12946 hybrid_clock.cc:648] HybridClock initialized: now 1790184277634187 us; error 38 us; skew 500 ppm
Sep 23 17:24:37 dist-test-slave-ptl9 krb5kdc[12705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184277, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:37.637234 12946 init.cc:377] Logged in from keytab as kudu/127.11.229.2@KRBTEST.COM (short username kudu)
I20260923 17:24:37.638221 12946 webserver.cc:460] Webserver started at http://127.11.229.2:36021/ using document root <none> and password file <none>
I20260923 17:24:37.638837 12946 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:37.638931 12946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:37.639190 12946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:37.641420 12946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-1/data/instance:
uuid: "39e5f9e0f62b4c88bd6f1bd4025ea688"
format_stamp: "Formatted at 2026-09-23 17:24:37 on dist-test-slave-ptl9"
server_key: "788947520c6df94acb5663a7a493a666"
server_key_iv: "f633165e673759b78aa9201d10420875"
server_key_version: "encryptionkey@0"
I20260923 17:24:37.642035 12946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance:
uuid: "39e5f9e0f62b4c88bd6f1bd4025ea688"
format_stamp: "Formatted at 2026-09-23 17:24:37 on dist-test-slave-ptl9"
server_key: "788947520c6df94acb5663a7a493a666"
server_key_iv: "f633165e673759b78aa9201d10420875"
server_key_version: "encryptionkey@0"
I20260923 17:24:37.646523 12946 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260923 17:24:37.649175 12961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:37.650681 12946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:24:37.650846 12946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-1/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-1/wal
uuid: "39e5f9e0f62b4c88bd6f1bd4025ea688"
format_stamp: "Formatted at 2026-09-23 17:24:37 on dist-test-slave-ptl9"
server_key: "788947520c6df94acb5663a7a493a666"
server_key_iv: "f633165e673759b78aa9201d10420875"
server_key_version: "encryptionkey@0"
I20260923 17:24:37.650980 12946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:37.662899 12946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:37.666244 12946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:37.666447 12946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:37.667062 12946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:24:37.667892 12946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:24:37.667963 12946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:37.668051 12946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:24:37.668100 12946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:37.677074 12946 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.2:36615
I20260923 17:24:37.677137 13074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.2:36615 every 8 connection(s)
I20260923 17:24:37.678133 12946 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
I20260923 17:24:37.680182 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 12946
I20260923 17:24:37.680292 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance
I20260923 17:24:37.680570 12180 external_mini_cluster.cc:1428] Setting key 52a36d782647d360e17c498d8eb98c4c
Sep 23 17:24:37 dist-test-slave-ptl9 krb5kdc[12705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184277, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.229.3@KRBTEST.COM; defaulting to no policy
I20260923 17:24:37.692497 12789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:37.680064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:58115 (local address 127.11.229.62:45773)
0923 17:24:37.680230 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:37.680233 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:37.681464 (+ 1231us) server_negotiation.cc:407] Connection header received
0923 17:24:37.682636 (+ 1172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:37.682641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:37.682697 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:37.682796 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:37.684032 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:37.684597 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:37.685851 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:37.686086 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:37.686198 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:37.688759 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:37.688763 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:37.688765 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:37.688788 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:37.690311 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:37.690876 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:37.690881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:37.690883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:37.690966 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:37.691320 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:37.691325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:37.691327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:37.691537 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:37.691668 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:37.692186 (+ 518us) server_negotiation.cc:299] Negotiation successful
0923 17:24:37.692322 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.3@KRBTEST.COM" created.
I20260923 17:24:37.693451 13077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:37.680641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:45773 (local address 127.11.229.2:58115)
0923 17:24:37.681288 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:37.681350 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:37.682416 (+ 1066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:37.682953 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:37.682960 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:37.683333 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:37.683885 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:37.683900 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:37.684733 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:37.684736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:37.685727 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:37.685736 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:37.686253 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:37.686257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:37.686423 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:37.687132 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:37.687150 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:37.688579 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:37.690424 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:37.690430 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:37.690447 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:37.690773 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:37.691061 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:37.691064 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:37.691066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:37.691222 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:37.691667 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:37.691675 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:37.691936 (+ 261us) client_negotiation.cc:769] Sending connection context
0923 17:24:37.692138 (+ 202us) client_negotiation.cc:240] Negotiation successful
0923 17:24:37.692475 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":149,"threads_started":1}
I20260923 17:24:37.694829 13075 heartbeater.cc:347] Connected to a master server at 127.11.229.62:45773
I20260923 17:24:37.695137 13075 heartbeater.cc:464] Registering TS with master...
I20260923 17:24:37.695760 13075 heartbeater.cc:511] Master 127.11.229.62:45773 requested a full tablet report, sending...
I20260923 17:24:37.697013 12750 ts_manager.cc:194] Registered new tserver with Master: 39e5f9e0f62b4c88bd6f1bd4025ea688 (127.11.229.2:36615)
I20260923 17:24:37.697667 12750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.2@KRBTEST.COM'} at 127.11.229.2:58115
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab.
Entry for principal kudu/127.11.229.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab.
I20260923 17:24:37.712759 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.3:0
--local_ip_for_outbound_sockets=127.11.229.3
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:45773
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:38607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:37.833993 13080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:37.834288 13080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:37.834336 13080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:37.838574 13080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:37.838647 13080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:37.838713 13080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.3
I20260923 17:24:37.844236 13080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:38607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:45773
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.3
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:37.845456 13080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:37.846678 13080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:37.857029 13085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:37.857079 13088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:37.857066 13080 server_base.cc:1034] running on GCE node
W20260923 17:24:37.857038 13086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:37.857813 13080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:37.858485 13080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:37.859689 13080 hybrid_clock.cc:648] HybridClock initialized: now 1790184277859651 us; error 50 us; skew 500 ppm
Sep 23 17:24:37 dist-test-slave-ptl9 krb5kdc[12705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184277, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:37.862643 13080 init.cc:377] Logged in from keytab as kudu/127.11.229.3@KRBTEST.COM (short username kudu)
I20260923 17:24:37.863562 13080 webserver.cc:460] Webserver started at http://127.11.229.3:38025/ using document root <none> and password file <none>
I20260923 17:24:37.864140 13080 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:37.864204 13080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:37.864374 13080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:37.866575 13080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-2/data/instance:
uuid: "a4fddac3829545b18326ed207041d322"
format_stamp: "Formatted at 2026-09-23 17:24:37 on dist-test-slave-ptl9"
server_key: "d1d6fbb1f43212b23f97374673d89814"
server_key_iv: "487a9b79e58fd739ea9a3f44e621a550"
server_key_version: "encryptionkey@0"
I20260923 17:24:37.867192 13080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance:
uuid: "a4fddac3829545b18326ed207041d322"
format_stamp: "Formatted at 2026-09-23 17:24:37 on dist-test-slave-ptl9"
server_key: "d1d6fbb1f43212b23f97374673d89814"
server_key_iv: "487a9b79e58fd739ea9a3f44e621a550"
server_key_version: "encryptionkey@0"
I20260923 17:24:37.871304 13080 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:24:37.873950 13095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:37.875233 13080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:24:37.875378 13080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-2/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-2/wal
uuid: "a4fddac3829545b18326ed207041d322"
format_stamp: "Formatted at 2026-09-23 17:24:37 on dist-test-slave-ptl9"
server_key: "d1d6fbb1f43212b23f97374673d89814"
server_key_iv: "487a9b79e58fd739ea9a3f44e621a550"
server_key_version: "encryptionkey@0"
I20260923 17:24:37.875501 13080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:37.897024 13080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:37.900490 13080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:37.900707 13080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:37.901378 13080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:24:37.902236 13080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:24:37.902315 13080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:37.902392 13080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:24:37.902446 13080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:37.911536 13080 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.3:36581
I20260923 17:24:37.911558 13208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.3:36581 every 8 connection(s)
I20260923 17:24:37.912438 13080 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
I20260923 17:24:37.920009 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 13080
I20260923 17:24:37.920140 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance
I20260923 17:24:37.920428 12180 external_mini_cluster.cc:1428] Setting key fbfcd19bde18389815bd1d6c59f2b23e
Sep 23 17:24:37 dist-test-slave-ptl9 krb5kdc[12705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184277, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:24:37.926433 12789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:37.914415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:55937 (local address 127.11.229.62:45773)
0923 17:24:37.914582 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:37.914587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:37.915581 (+ 994us) server_negotiation.cc:407] Connection header received
0923 17:24:37.916507 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:37.916510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:37.916559 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:37.916637 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:37.917916 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:37.918450 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:37.919541 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:37.919789 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:37.919908 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:37.922801 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:37.922804 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:37.922807 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:37.922837 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:37.924483 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:37.925039 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:37.925042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:37.925044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:37.925130 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:37.925497 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:37.925499 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:37.925501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:37.925687 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:37.925763 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:37.926215 (+ 452us) server_negotiation.cc:299] Negotiation successful
0923 17:24:37.926312 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:24:37.927290 13211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:37.914866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:45773 (local address 127.11.229.3:55937)
0923 17:24:37.915419 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:37.915473 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:37.916284 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:37.916805 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:37.916812 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:37.917163 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:37.917779 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:37.917795 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:37.918556 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:37.918558 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:37.919435 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:37.919444 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:37.919962 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:37.919965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:37.920207 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:37.920885 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:37.920901 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:37.922647 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:37.924622 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:37.924631 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:37.924648 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:37.924948 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:37.925213 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:37.925216 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:37.925217 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:37.925419 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:37.925775 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:37.925780 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:37.926004 (+ 224us) client_negotiation.cc:769] Sending connection context
0923 17:24:37.926152 (+ 148us) client_negotiation.cc:240] Negotiation successful
0923 17:24:37.926394 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":111,"threads_started":1}
I20260923 17:24:37.928622 13209 heartbeater.cc:347] Connected to a master server at 127.11.229.62:45773
I20260923 17:24:37.928889 13209 heartbeater.cc:464] Registering TS with master...
I20260923 17:24:37.929495 13209 heartbeater.cc:511] Master 127.11.229.62:45773 requested a full tablet report, sending...
I20260923 17:24:37.930581 12750 ts_manager.cc:194] Registered new tserver with Master: a4fddac3829545b18326ed207041d322 (127.11.229.3:36581)
I20260923 17:24:37.931135 12750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.3@KRBTEST.COM'} at 127.11.229.3:55937
I20260923 17:24:37.934567 12180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:24:37.946127 12789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:37.938309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35536 (local address 127.11.229.62:45773)
0923 17:24:37.938448 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:37.938452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:37.938638 (+ 186us) server_negotiation.cc:407] Connection header received
0923 17:24:37.938741 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:37.938743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:37.938785 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:37.938854 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:37.939515 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:37.940040 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:37.940944 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:37.941166 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:37.941249 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:37.942791 (+ 1542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:37.942793 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:37.942795 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:37.942813 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:37.944208 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:37.944649 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:37.944651 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:37.944653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:37.944727 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:37.945091 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:37.945093 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:37.945094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:37.945391 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:37.945501 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:37.945818 (+ 317us) server_negotiation.cc:299] Negotiation successful
0923 17:24:37.945978 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 17:24:37.951453 12750 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:24:37.954758 12750 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:24:37.970260 13225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:37.965136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54686 (local address 127.11.229.1:36031)
0923 17:24:37.965447 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:37.965464 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:37.965485 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:24:37.965563 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:37.965568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:37.965769 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:37.965905 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:24:37.966219 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:37.967309 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:37.968632 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:37.969511 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:37.969637 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:37.969812 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:37.969979 (+ 167us) server_negotiation.cc:299] Negotiation successful
0923 17:24:37.970087 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":79,"threads_started":1}
I20260923 17:24:37.970278 13220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:37.963629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:36031 (local address 127.0.0.1:54686)
0923 17:24:37.964310 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:37.964324 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:37.964405 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:37.965906 (+ 1501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:37.965911 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:37.965929 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:24:37.966008 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:37.966014 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:37.967457 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:37.967460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:37.968501 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:37.968510 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:37.969657 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:37.969661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:37.969772 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:37.969815 (+ 43us) client_negotiation.cc:769] Sending connection context
0923 17:24:37.969934 (+ 119us) client_negotiation.cc:240] Negotiation successful
0923 17:24:37.970061 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"mutex_wait_us":66,"thread_start_us":89,"threads_started":1}
I20260923 17:24:37.970634 13222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:37.963767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48786 (local address 127.11.229.3:36581)
0923 17:24:37.964533 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:37.964547 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:37.964570 (+ 23us) server_negotiation.cc:407] Connection header received
0923 17:24:37.964646 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:37.964651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:37.964874 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:37.965066 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:24:37.965449 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:37.966761 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:37.968813 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:37.969630 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:37.969721 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:37.969827 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:37.970354 (+ 527us) server_negotiation.cc:299] Negotiation successful
0923 17:24:37.970478 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":141,"threads_started":1}
I20260923 17:24:37.970669 13223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:37.964363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:36615 (local address 127.0.0.1:45266)
0923 17:24:37.964919 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:37.964932 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:37.965017 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:37.965686 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:37.965689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:37.965706 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:24:37.965812 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:37.965820 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:37.967059 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:37.967062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:37.968683 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:37.968690 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:37.970262 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:37.970265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:37.970374 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:37.970401 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 17:24:37.970455 (+ 54us) client_negotiation.cc:240] Negotiation successful
0923 17:24:37.970524 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":233,"thread_start_us":85,"threads_started":1}
I20260923 17:24:37.970911 13221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:37.963594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:36581 (local address 127.0.0.1:48786)
0923 17:24:37.964006 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:37.964071 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:37.964232 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:37.965059 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:37.965065 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:37.965089 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:24:37.965272 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:37.965319 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:37.966942 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:37.966945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:37.968678 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:37.968690 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:37.969898 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:37.969925 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:37.970038 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:37.970064 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 17:24:37.970137 (+ 73us) client_negotiation.cc:240] Negotiation successful
0923 17:24:37.970204 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":195,"thread_start_us":88,"threads_started":1}
I20260923 17:24:37.971388 13224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:37.964732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45266 (local address 127.11.229.2:36615)
0923 17:24:37.965305 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:37.965317 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:37.965338 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:24:37.965410 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:37.965414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:37.965550 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:37.965709 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:24:37.965946 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:37.966873 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:37.969378 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:37.970129 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:37.970957 (+ 828us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:37.971131 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:37.971183 (+ 52us) server_negotiation.cc:299] Negotiation successful
0923 17:24:37.971248 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":119,"threads_started":1}
I20260923 17:24:37.974010 13143 tablet_service.cc:1467] Processing CreateTablet for tablet a4f07dde9a5a47c4b8f7f292a4423dac (DEFAULT_TABLE table=test-table [id=5b20ddbe7ef44d42a74f363ad84d96ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:24:37.974210 12875 tablet_service.cc:1467] Processing CreateTablet for tablet a4f07dde9a5a47c4b8f7f292a4423dac (DEFAULT_TABLE table=test-table [id=5b20ddbe7ef44d42a74f363ad84d96ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:24:37.974225 13009 tablet_service.cc:1467] Processing CreateTablet for tablet a4f07dde9a5a47c4b8f7f292a4423dac (DEFAULT_TABLE table=test-table [id=5b20ddbe7ef44d42a74f363ad84d96ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:24:37.975136 13143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4f07dde9a5a47c4b8f7f292a4423dac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:37.975248 13009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4f07dde9a5a47c4b8f7f292a4423dac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:37.975247 12875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4f07dde9a5a47c4b8f7f292a4423dac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:37.982697 13227 tablet_bootstrap.cc:492] T a4f07dde9a5a47c4b8f7f292a4423dac P 39e5f9e0f62b4c88bd6f1bd4025ea688: Bootstrap starting.
I20260923 17:24:37.983088 13226 tablet_bootstrap.cc:492] T a4f07dde9a5a47c4b8f7f292a4423dac P a4fddac3829545b18326ed207041d322: Bootstrap starting.
I20260923 17:24:37.985029 13226 tablet_bootstrap.cc:654] T a4f07dde9a5a47c4b8f7f292a4423dac P a4fddac3829545b18326ed207041d322: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:37.985419 13227 tablet_bootstrap.cc:654] T a4f07dde9a5a47c4b8f7f292a4423dac P 39e5f9e0f62b4c88bd6f1bd4025ea688: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:37.985509 13228 tablet_bootstrap.cc:492] T a4f07dde9a5a47c4b8f7f292a4423dac P 86c5cc4be68e46e5977830c35abed64a: Bootstrap starting.
I20260923 17:24:37.986469 13227 log.cc:824] T a4f07dde9a5a47c4b8f7f292a4423dac P 39e5f9e0f62b4c88bd6f1bd4025ea688: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:37.986469 13226 log.cc:824] T a4f07dde9a5a47c4b8f7f292a4423dac P a4fddac3829545b18326ed207041d322: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:37.987623 13228 tablet_bootstrap.cc:654] T a4f07dde9a5a47c4b8f7f292a4423dac P 86c5cc4be68e46e5977830c35abed64a: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:37.988379 13228 log.cc:824] T a4f07dde9a5a47c4b8f7f292a4423dac P 86c5cc4be68e46e5977830c35abed64a: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:37.988667 13227 tablet_bootstrap.cc:492] T a4f07dde9a5a47c4b8f7f292a4423dac P 39e5f9e0f62b4c88bd6f1bd4025ea688: No bootstrap required, opened a new log
I20260923 17:24:37.988685 13226 tablet_bootstrap.cc:492] T a4f07dde9a5a47c4b8f7f292a4423dac P a4fddac3829545b18326ed207041d322: No bootstrap required, opened a new log
I20260923 17:24:37.988878 13226 ts_tablet_manager.cc:1397] T a4f07dde9a5a47c4b8f7f292a4423dac P a4fddac3829545b18326ed207041d322: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 17:24:37.988875 13227 ts_tablet_manager.cc:1397] T a4f07dde9a5a47c4b8f7f292a4423dac P 39e5f9e0f62b4c88bd6f1bd4025ea688: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 17:24:37.990377 13228 tablet_bootstrap.cc:492] T a4f07dde9a5a47c4b8f7f292a4423dac P 86c5cc4be68e46e5977830c35abed64a: No bootstrap required, opened a new log
I20260923 17:24:37.990581 13228 ts_tablet_manager.cc:1397] T a4f07dde9a5a47c4b8f7f292a4423dac P 86c5cc4be68e46e5977830c35abed64a: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260923 17:24:37.992375 13227 raft_consensus.cc:348] T a4f07dde9a5a47c4b8f7f292a4423dac P 39e5f9e0f62b4c88bd6f1bd4025ea688 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fddac3829545b18326ed207041d322" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 36581 } } peers { permanent_uuid: "86c5cc4be68e46e5977830c35abed64a" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36031 } } peers { permanent_uuid: "39e5f9e0f62b4c88bd6f1bd4025ea688" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36615 } }
I20260923 17:24:37.992695 13227 raft_consensus.cc:374] T a4f07dde9a5a47c4b8f7f292a4423dac P 39e5f9e0f62b4c88bd6f1bd4025ea688 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:37.992777 13227 raft_consensus.cc:729] T a4f07dde9a5a47c4b8f7f292a4423dac P 39e5f9e0f62b4c88bd6f1bd4025ea688 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39e5f9e0f62b4c88bd6f1bd4025ea688, State: Initialized, Role: FOLLOWER
I20260923 17:24:37.993340 13227 consensus_queue.cc:260] T a4f07dde9a5a47c4b8f7f292a4423dac P 39e5f9e0f62b4c88bd6f1bd4025ea688 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fddac3829545b18326ed207041d322" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 36581 } } peers { permanent_uuid: "86c5cc4be68e46e5977830c35abed64a" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36031 } } peers { permanent_uuid: "39e5f9e0f62b4c88bd6f1bd4025ea688" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36615 } }
I20260923 17:24:37.993341 13226 raft_consensus.cc:348] T a4f07dde9a5a47c4b8f7f292a4423dac P a4fddac3829545b18326ed207041d322 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fddac3829545b18326ed207041d322" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 36581 } } peers { permanent_uuid: "86c5cc4be68e46e5977830c35abed64a" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36031 } } peers { permanent_uuid: "39e5f9e0f62b4c88bd6f1bd4025ea688" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36615 } }
I20260923 17:24:37.993613 13226 raft_consensus.cc:374] T a4f07dde9a5a47c4b8f7f292a4423dac P a4fddac3829545b18326ed207041d322 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:37.993671 13226 raft_consensus.cc:729] T a4f07dde9a5a47c4b8f7f292a4423dac P a4fddac3829545b18326ed207041d322 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4fddac3829545b18326ed207041d322, State: Initialized, Role: FOLLOWER
I20260923 17:24:37.994091 13226 consensus_queue.cc:260] T a4f07dde9a5a47c4b8f7f292a4423dac P a4fddac3829545b18326ed207041d322 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fddac3829545b18326ed207041d322" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 36581 } } peers { permanent_uuid: "86c5cc4be68e46e5977830c35abed64a" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36031 } } peers { permanent_uuid: "39e5f9e0f62b4c88bd6f1bd4025ea688" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36615 } }
I20260923 17:24:37.994365 13227 ts_tablet_manager.cc:1428] T a4f07dde9a5a47c4b8f7f292a4423dac P 39e5f9e0f62b4c88bd6f1bd4025ea688: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260923 17:24:37.994788 13209 heartbeater.cc:503] Master 127.11.229.62:45773 was elected leader, sending a full tablet report...
I20260923 17:24:37.995083 13226 ts_tablet_manager.cc:1428] T a4f07dde9a5a47c4b8f7f292a4423dac P a4fddac3829545b18326ed207041d322: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260923 17:24:37.995046 13228 raft_consensus.cc:348] T a4f07dde9a5a47c4b8f7f292a4423dac P 86c5cc4be68e46e5977830c35abed64a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fddac3829545b18326ed207041d322" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 36581 } } peers { permanent_uuid: "86c5cc4be68e46e5977830c35abed64a" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36031 } } peers { permanent_uuid: "39e5f9e0f62b4c88bd6f1bd4025ea688" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36615 } }
I20260923 17:24:37.995292 13228 raft_consensus.cc:374] T a4f07dde9a5a47c4b8f7f292a4423dac P 86c5cc4be68e46e5977830c35abed64a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:37.995361 13228 raft_consensus.cc:729] T a4f07dde9a5a47c4b8f7f292a4423dac P 86c5cc4be68e46e5977830c35abed64a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86c5cc4be68e46e5977830c35abed64a, State: Initialized, Role: FOLLOWER
I20260923 17:24:37.995734 13228 consensus_queue.cc:260] T a4f07dde9a5a47c4b8f7f292a4423dac P 86c5cc4be68e46e5977830c35abed64a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fddac3829545b18326ed207041d322" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 36581 } } peers { permanent_uuid: "86c5cc4be68e46e5977830c35abed64a" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36031 } } peers { permanent_uuid: "39e5f9e0f62b4c88bd6f1bd4025ea688" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36615 } }
I20260923 17:24:37.995982 13075 heartbeater.cc:503] Master 127.11.229.62:45773 was elected leader, sending a full tablet report...
I20260923 17:24:37.996371 12941 heartbeater.cc:503] Master 127.11.229.62:45773 was elected leader, sending a full tablet report...
I20260923 17:24:37.996733 13228 ts_tablet_manager.cc:1428] T a4f07dde9a5a47c4b8f7f292a4423dac P 86c5cc4be68e46e5977830c35abed64a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260923 17:24:38.163470 13210 tablet.cc:2369] T a4f07dde9a5a47c4b8f7f292a4423dac P a4fddac3829545b18326ed207041d322: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:24:38.179330 13076 tablet.cc:2369] T a4f07dde9a5a47c4b8f7f292a4423dac P 39e5f9e0f62b4c88bd6f1bd4025ea688: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:24:38.208760 12942 tablet.cc:2369] T a4f07dde9a5a47c4b8f7f292a4423dac P 86c5cc4be68e46e5977830c35abed64a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:24:38.344718 13233 raft_consensus.cc:482] T a4f07dde9a5a47c4b8f7f292a4423dac P a4fddac3829545b18326ed207041d322 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:24:38.345038 13233 raft_consensus.cc:504] T a4f07dde9a5a47c4b8f7f292a4423dac P a4fddac3829545b18326ed207041d322 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fddac3829545b18326ed207041d322" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 36581 } } peers { permanent_uuid: "86c5cc4be68e46e5977830c35abed64a" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36031 } } peers { permanent_uuid: "39e5f9e0f62b4c88bd6f1bd4025ea688" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36615 } }
I20260923 17:24:38.346372 13233 leader_election.cc:288] T a4f07dde9a5a47c4b8f7f292a4423dac P a4fddac3829545b18326ed207041d322 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86c5cc4be68e46e5977830c35abed64a (127.11.229.1:36031), 39e5f9e0f62b4c88bd6f1bd4025ea688 (127.11.229.2:36615)
I20260923 17:24:38.351449 13225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:38.346804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:59193 (local address 127.11.229.1:36031)
0923 17:24:38.347067 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:38.347072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:38.347092 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:24:38.347156 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:38.347160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:38.347210 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:38.347298 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:24:38.347644 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:38.348478 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:38.350104 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:38.350728 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:38.350860 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:38.350914 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:38.351242 (+ 328us) server_negotiation.cc:299] Negotiation successful
0923 17:24:38.351316 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260923 17:24:38.351449 13224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:38.347064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:35183 (local address 127.11.229.2:36615)
0923 17:24:38.347200 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:38.347207 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:38.347435 (+ 228us) server_negotiation.cc:407] Connection header received
0923 17:24:38.347643 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:38.347646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:38.347699 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:38.347777 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:24:38.347987 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:38.348683 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:38.350104 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:38.350728 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:38.350860 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:38.350914 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:38.351242 (+ 328us) server_negotiation.cc:299] Negotiation successful
0923 17:24:38.351316 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 17:24:38.351464 13211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:38.346638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:36031 (local address 127.11.229.3:59193)
0923 17:24:38.346792 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:38.346810 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:38.346924 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:38.347314 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:38.347317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:38.347338 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:24:38.347514 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:38.347521 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:38.348605 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:38.348607 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:38.349932 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:38.349944 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:38.350872 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:38.350876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:38.351075 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:38.351124 (+ 49us) client_negotiation.cc:769] Sending connection context
0923 17:24:38.351202 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 17:24:38.351325 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":107}
I20260923 17:24:38.351471 13235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:38.346951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:36615 (local address 127.11.229.3:35183)
0923 17:24:38.347340 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:38.347353 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:38.347514 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:38.347780 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:38.347781 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:38.347794 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:24:38.347880 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:38.347884 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:38.348786 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:38.348789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:38.349957 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:38.349963 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:38.350872 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:38.350876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:38.351087 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:38.351120 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 17:24:38.351202 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 17:24:38.351306 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":82,"thread_start_us":150,"threads_started":1}
I20260923 17:24:38.352355 13029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4f07dde9a5a47c4b8f7f292a4423dac" candidate_uuid: "a4fddac3829545b18326ed207041d322" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39e5f9e0f62b4c88bd6f1bd4025ea688" is_pre_election: true
I20260923 17:24:38.352358 12895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4f07dde9a5a47c4b8f7f292a4423dac" candidate_uuid: "a4fddac3829545b18326ed207041d322" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86c5cc4be68e46e5977830c35abed64a" is_pre_election: true
I20260923 17:24:38.352696 13029 raft_consensus.cc:2454] T a4f07dde9a5a47c4b8f7f292a4423dac P 39e5f9e0f62b4c88bd6f1bd4025ea688 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4fddac3829545b18326ed207041d322 in term 0.
I20260923 17:24:38.352696 12895 raft_consensus.cc:2454] T a4f07dde9a5a47c4b8f7f292a4423dac P 86c5cc4be68e46e5977830c35abed64a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4fddac3829545b18326ed207041d322 in term 0.
I20260923 17:24:38.353277 13098 leader_election.cc:302] T a4f07dde9a5a47c4b8f7f292a4423dac P a4fddac3829545b18326ed207041d322 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39e5f9e0f62b4c88bd6f1bd4025ea688, a4fddac3829545b18326ed207041d322; no voters:
I20260923 17:24:38.353543 13233 raft_consensus.cc:2781] T a4f07dde9a5a47c4b8f7f292a4423dac P a4fddac3829545b18326ed207041d322 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:24:38.353637 13233 raft_consensus.cc:482] T a4f07dde9a5a47c4b8f7f292a4423dac P a4fddac3829545b18326ed207041d322 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:24:38.353682 13233 raft_consensus.cc:3037] T a4f07dde9a5a47c4b8f7f292a4423dac P a4fddac3829545b18326ed207041d322 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:38.354895 13233 raft_consensus.cc:504] T a4f07dde9a5a47c4b8f7f292a4423dac P a4fddac3829545b18326ed207041d322 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fddac3829545b18326ed207041d322" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 36581 } } peers { permanent_uuid: "86c5cc4be68e46e5977830c35abed64a" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36031 } } peers { permanent_uuid: "39e5f9e0f62b4c88bd6f1bd4025ea688" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36615 } }
I20260923 17:24:38.355288 13233 leader_election.cc:288] T a4f07dde9a5a47c4b8f7f292a4423dac P a4fddac3829545b18326ed207041d322 [CANDIDATE]: Term 1 election: Requested vote from peers 86c5cc4be68e46e5977830c35abed64a (127.11.229.1:36031), 39e5f9e0f62b4c88bd6f1bd4025ea688 (127.11.229.2:36615)
I20260923 17:24:38.355854 12895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4f07dde9a5a47c4b8f7f292a4423dac" candidate_uuid: "a4fddac3829545b18326ed207041d322" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86c5cc4be68e46e5977830c35abed64a"
I20260923 17:24:38.355903 13029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4f07dde9a5a47c4b8f7f292a4423dac" candidate_uuid: "a4fddac3829545b18326ed207041d322" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39e5f9e0f62b4c88bd6f1bd4025ea688"
I20260923 17:24:38.355999 12895 raft_consensus.cc:3037] T a4f07dde9a5a47c4b8f7f292a4423dac P 86c5cc4be68e46e5977830c35abed64a [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:38.356012 13029 raft_consensus.cc:3037] T a4f07dde9a5a47c4b8f7f292a4423dac P 39e5f9e0f62b4c88bd6f1bd4025ea688 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:38.357448 13029 raft_consensus.cc:2454] T a4f07dde9a5a47c4b8f7f292a4423dac P 39e5f9e0f62b4c88bd6f1bd4025ea688 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4fddac3829545b18326ed207041d322 in term 1.
I20260923 17:24:38.357448 12895 raft_consensus.cc:2454] T a4f07dde9a5a47c4b8f7f292a4423dac P 86c5cc4be68e46e5977830c35abed64a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4fddac3829545b18326ed207041d322 in term 1.
I20260923 17:24:38.357791 13098 leader_election.cc:302] T a4f07dde9a5a47c4b8f7f292a4423dac P a4fddac3829545b18326ed207041d322 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86c5cc4be68e46e5977830c35abed64a, a4fddac3829545b18326ed207041d322; no voters:
I20260923 17:24:38.357956 13233 raft_consensus.cc:2781] T a4f07dde9a5a47c4b8f7f292a4423dac P a4fddac3829545b18326ed207041d322 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:24:38.358217 13233 raft_consensus.cc:686] T a4f07dde9a5a47c4b8f7f292a4423dac P a4fddac3829545b18326ed207041d322 [term 1 LEADER]: Becoming Leader. State: Replica: a4fddac3829545b18326ed207041d322, State: Running, Role: LEADER
I20260923 17:24:38.358633 13233 consensus_queue.cc:237] T a4f07dde9a5a47c4b8f7f292a4423dac P a4fddac3829545b18326ed207041d322 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fddac3829545b18326ed207041d322" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 36581 } } peers { permanent_uuid: "86c5cc4be68e46e5977830c35abed64a" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36031 } } peers { permanent_uuid: "39e5f9e0f62b4c88bd6f1bd4025ea688" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36615 } }
I20260923 17:24:38.362031 12748 catalog_manager.cc:5495] T a4f07dde9a5a47c4b8f7f292a4423dac P a4fddac3829545b18326ed207041d322 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4fddac3829545b18326ed207041d322 (127.11.229.3). New cstate: current_term: 1 leader_uuid: "a4fddac3829545b18326ed207041d322" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fddac3829545b18326ed207041d322" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 36581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86c5cc4be68e46e5977830c35abed64a" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39e5f9e0f62b4c88bd6f1bd4025ea688" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36615 } health_report { overall_health: UNKNOWN } } }
Sep 23 17:24:38 dist-test-slave-ptl9 krb5kdc[12705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.1@KRBTEST.COM
I20260923 17:24:38.429359 13225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:38.420735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54688 (local address 127.11.229.1:36031)
0923 17:24:38.421007 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:38.421011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:38.421025 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:24:38.421067 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:38.421069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:38.421112 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:38.421180 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:38.421862 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:38.422360 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:38.423158 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:38.423367 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:38.423438 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:38.425990 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:38.426004 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:38.426019 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:38.426050 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:38.427783 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:38.428165 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:38.428175 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:38.428177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:38.428274 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:38.428503 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:38.428505 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:38.428506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:38.428762 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:38.428915 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:38.429064 (+ 149us) server_negotiation.cc:299] Negotiation successful
0923 17:24:38.429159 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
Sep 23 17:24:38 dist-test-slave-ptl9 krb5kdc[12705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184278, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 17:24:38 dist-test-slave-ptl9 krb5kdc[12705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184278, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.229.1@KRBTEST.COM
I20260923 17:24:38.457000 13225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:38.448168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54696 (local address 127.11.229.1:36031)
0923 17:24:38.448345 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:38.448350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:38.448483 (+ 133us) server_negotiation.cc:407] Connection header received
0923 17:24:38.448688 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:38.448691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:38.448735 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:38.448824 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:38.449571 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:38.450112 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:38.450959 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:38.451200 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:38.451312 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:38.453780 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:38.453784 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:38.453786 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:38.453811 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:38.455235 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:38.455653 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:38.455656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:38.455657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:38.455731 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:38.455978 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:38.455980 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:38.455981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:38.456141 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:38.456237 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:38.456620 (+ 383us) server_negotiation.cc:299] Negotiation successful
0923 17:24:38.456733 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
Sep 23 17:24:38 dist-test-slave-ptl9 krb5kdc[12705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 17:24:38 dist-test-slave-ptl9 krb5kdc[12705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.1@KRBTEST.COM
I20260923 17:24:38.486316 13225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:38.476834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54708 (local address 127.11.229.1:36031)
0923 17:24:38.477098 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:38.477104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:38.477182 (+ 78us) server_negotiation.cc:407] Connection header received
0923 17:24:38.477409 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:38.477412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:38.477460 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:38.477533 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:38.478478 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:38.479226 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:38.480316 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:38.480525 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:38.480629 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:38.483115 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:38.483118 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:38.483120 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:38.483146 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:38.484565 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:38.484994 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:38.484998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:38.485000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:38.485088 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:38.485442 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:38.485445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:38.485447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:38.485623 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:38.485764 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:38.486013 (+ 249us) server_negotiation.cc:299] Negotiation successful
0923 17:24:38.486147 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
W20260923 17:24:38.487679 12855 tablet_service.cc:2908] Rejecting scan request for tablet a4f07dde9a5a47c4b8f7f292a4423dac: Uninitialized: clean time has not yet been initialized
I20260923 17:24:38.490309 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 12812
I20260923 17:24:38.496790 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 12946
I20260923 17:24:38.502472 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 13080
I20260923 17:24:38.508082 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 12718
2026-09-23T17:24:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2734 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:24:38 dist-test-slave-ptl9 krb5kdc[13250](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:24:38 dist-test-slave-ptl9 krb5kdc[13250](info): set up 2 sockets
Sep 23 17:24:38 dist-test-slave-ptl9 krb5kdc[13250](info): commencing operation
krb5kdc: starting...
W20260923 17:24:39.541246 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:24:39 dist-test-slave-ptl9 krb5kdc[13250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:24:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:24:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.229.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
I20260923 17:24:39.669395 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:40149
--webserver_interface=127.11.229.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:39605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:40149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:39.785053 13263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:39.785413 13263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:39.785501 13263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:39.789650 13263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:24:39.789757 13263 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 17:24:39.789788 13263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:39.789841 13263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:39.789903 13263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:24:39.789949 13263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:24:39.795629 13263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:39605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:40149
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:40149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:39.796886 13263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:39.798089 13263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:39.807437 13269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:39.807413 13268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:39.807597 13263 server_base.cc:1034] running on GCE node
W20260923 17:24:39.807396 13271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:39.808492 13263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:39.809659 13263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:39.810823 13263 hybrid_clock.cc:648] HybridClock initialized: now 1790184279810809 us; error 28 us; skew 500 ppm
Sep 23 17:24:39 dist-test-slave-ptl9 krb5kdc[13250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184279, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:39.813776 13263 init.cc:377] Logged in from keytab as kudu/127.11.229.62@KRBTEST.COM (short username kudu)
I20260923 17:24:39.814702 13263 webserver.cc:460] Webserver started at http://127.11.229.62:39683/ using document root <none> and password file <none>
I20260923 17:24:39.815290 13263 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:39.815388 13263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:39.815641 13263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:39.817823 13263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "9c5f394982af4a14b71ea8ab5e55c65f"
format_stamp: "Formatted at 2026-09-23 17:24:39 on dist-test-slave-ptl9"
server_key: "40652a38b1477b57ffc806d70d21b9a3"
server_key_iv: "b37eaf9d31e63e565b628a9ff9ae1fc1"
server_key_version: "encryptionkey@0"
I20260923 17:24:39.818400 13263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "9c5f394982af4a14b71ea8ab5e55c65f"
format_stamp: "Formatted at 2026-09-23 17:24:39 on dist-test-slave-ptl9"
server_key: "40652a38b1477b57ffc806d70d21b9a3"
server_key_iv: "b37eaf9d31e63e565b628a9ff9ae1fc1"
server_key_version: "encryptionkey@0"
I20260923 17:24:39.822461 13263 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260923 17:24:39.825035 13278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:39.826427 13263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:24:39.826586 13263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "9c5f394982af4a14b71ea8ab5e55c65f"
format_stamp: "Formatted at 2026-09-23 17:24:39 on dist-test-slave-ptl9"
server_key: "40652a38b1477b57ffc806d70d21b9a3"
server_key_iv: "b37eaf9d31e63e565b628a9ff9ae1fc1"
server_key_version: "encryptionkey@0"
I20260923 17:24:39.826709 13263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:39.852736 13263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:39.856417 13263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:39.856634 13263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:39.865026 13263 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:40149
I20260923 17:24:39.865100 13340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:40149 every 8 connection(s)
I20260923 17:24:39.866217 13263 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:24:39.869231 13341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:39.875175 13341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c5f394982af4a14b71ea8ab5e55c65f: Bootstrap starting.
I20260923 17:24:39.876215 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 13263
I20260923 17:24:39.876312 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:24:39.876566 12180 external_mini_cluster.cc:1428] Setting key 6a4f00129b6d517dd5e22cfd270b9389
I20260923 17:24:39.878099 13341 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c5f394982af4a14b71ea8ab5e55c65f: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:39.878878 13341 log.cc:824] T 00000000000000000000000000000000 P 9c5f394982af4a14b71ea8ab5e55c65f: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:39.881153 13341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c5f394982af4a14b71ea8ab5e55c65f: No bootstrap required, opened a new log
I20260923 17:24:39.885483 13341 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c5f394982af4a14b71ea8ab5e55c65f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c5f394982af4a14b71ea8ab5e55c65f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 40149 } }
Sep 23 17:24:39 dist-test-slave-ptl9 krb5kdc[13250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:24:39.885779 13341 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c5f394982af4a14b71ea8ab5e55c65f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:39.885842 13341 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c5f394982af4a14b71ea8ab5e55c65f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c5f394982af4a14b71ea8ab5e55c65f, State: Initialized, Role: FOLLOWER
I20260923 17:24:39.886288 13341 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c5f394982af4a14b71ea8ab5e55c65f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c5f394982af4a14b71ea8ab5e55c65f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 40149 } }
I20260923 17:24:39.886462 13341 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c5f394982af4a14b71ea8ab5e55c65f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:24:39.886574 13341 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c5f394982af4a14b71ea8ab5e55c65f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:24:39.886690 13341 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c5f394982af4a14b71ea8ab5e55c65f [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:39.888005 13341 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c5f394982af4a14b71ea8ab5e55c65f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c5f394982af4a14b71ea8ab5e55c65f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 40149 } }
I20260923 17:24:39.888377 13341 leader_election.cc:302] T 00000000000000000000000000000000 P 9c5f394982af4a14b71ea8ab5e55c65f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c5f394982af4a14b71ea8ab5e55c65f; no voters:
I20260923 17:24:39.888767 13341 leader_election.cc:288] T 00000000000000000000000000000000 P 9c5f394982af4a14b71ea8ab5e55c65f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:24:39.888885 13346 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c5f394982af4a14b71ea8ab5e55c65f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:24:39.889107 13346 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c5f394982af4a14b71ea8ab5e55c65f [term 1 LEADER]: Becoming Leader. State: Replica: 9c5f394982af4a14b71ea8ab5e55c65f, State: Running, Role: LEADER
I20260923 17:24:39.889494 13346 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c5f394982af4a14b71ea8ab5e55c65f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c5f394982af4a14b71ea8ab5e55c65f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 40149 } }
I20260923 17:24:39.890237 13341 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c5f394982af4a14b71ea8ab5e55c65f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:24:39.891141 13348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c5f394982af4a14b71ea8ab5e55c65f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c5f394982af4a14b71ea8ab5e55c65f. Latest consensus state: current_term: 1 leader_uuid: "9c5f394982af4a14b71ea8ab5e55c65f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c5f394982af4a14b71ea8ab5e55c65f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 40149 } } }
I20260923 17:24:39.891280 13348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c5f394982af4a14b71ea8ab5e55c65f [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:39.891615 13347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c5f394982af4a14b71ea8ab5e55c65f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c5f394982af4a14b71ea8ab5e55c65f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c5f394982af4a14b71ea8ab5e55c65f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 40149 } } }
I20260923 17:24:39.891721 13347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c5f394982af4a14b71ea8ab5e55c65f [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:39.892057 13355 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:24:39.892916 13344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:39.877809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34696 (local address 127.11.229.62:40149)
0923 17:24:39.878273 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:39.878290 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:39.878327 (+ 37us) server_negotiation.cc:407] Connection header received
0923 17:24:39.878956 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:39.878972 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:39.879380 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:39.879692 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:39.880396 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:39.881869 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:39.882729 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:39.883532 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:39.883710 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:39.886388 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:39.886396 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:39.886416 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:39.886450 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:39.888553 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:39.889135 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:39.889141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:39.889148 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:39.889325 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:39.889694 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:39.889697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:39.889699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:39.890236 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:39.890403 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:39.890900 (+ 497us) server_negotiation.cc:299] Negotiation successful
0923 17:24:39.891288 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":134,"threads_started":1}
I20260923 17:24:39.895299 13355 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:24:39.901150 13355 catalog_manager.cc:1347] Generated new cluster ID: 01e62136769446088fe2cb6fdd92f093
I20260923 17:24:39.901257 13355 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:24:39.917898 13355 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:24:39.919055 13355 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:24:39.934454 13355 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c5f394982af4a14b71ea8ab5e55c65f: Generated new TSK 0
I20260923 17:24:39.935406 13355 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.229.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab.
Entry for principal kudu/127.11.229.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab.
I20260923 17:24:39.972078 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.1:0
--local_ip_for_outbound_sockets=127.11.229.1
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:40149
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:39605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:40.088287 13367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:40.088585 13367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:40.088663 13367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:40.093034 13367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:40.093102 13367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:40.093164 13367 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:24:40.093235 13367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.1
I20260923 17:24:40.098673 13367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:39605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.229.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.229.62:40149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.1
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:40.099829 13367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:40.100988 13367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:40.110841 13372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:40.111011 13367 server_base.cc:1034] running on GCE node
W20260923 17:24:40.110939 13373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:40.110848 13375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:40.111593 13367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:40.112272 13367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:40.113467 13367 hybrid_clock.cc:648] HybridClock initialized: now 1790184280113441 us; error 34 us; skew 500 ppm
Sep 23 17:24:40 dist-test-slave-ptl9 krb5kdc[13250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184280, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:40.116297 13367 init.cc:377] Logged in from keytab as kudu/127.11.229.1@KRBTEST.COM (short username kudu)
I20260923 17:24:40.117245 13367 webserver.cc:460] Webserver started at http://127.11.229.1:34617/ using document root <none> and password file <none>
I20260923 17:24:40.117851 13367 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:40.117909 13367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:40.118084 13367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:40.120146 13367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-0/data/instance:
uuid: "73dcd776a58041f0966e5c879ffc0281"
format_stamp: "Formatted at 2026-09-23 17:24:40 on dist-test-slave-ptl9"
server_key: "9b4e241f66527e8c5dad4d4d1200e75e"
server_key_iv: "c13086e9fd838c8c91d75c2680f34016"
server_key_version: "encryptionkey@0"
I20260923 17:24:40.120713 13367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance:
uuid: "73dcd776a58041f0966e5c879ffc0281"
format_stamp: "Formatted at 2026-09-23 17:24:40 on dist-test-slave-ptl9"
server_key: "9b4e241f66527e8c5dad4d4d1200e75e"
server_key_iv: "c13086e9fd838c8c91d75c2680f34016"
server_key_version: "encryptionkey@0"
I20260923 17:24:40.124526 13367 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260923 17:24:40.127069 13382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:40.128364 13367 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:24:40.128497 13367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-0/wal
uuid: "73dcd776a58041f0966e5c879ffc0281"
format_stamp: "Formatted at 2026-09-23 17:24:40 on dist-test-slave-ptl9"
server_key: "9b4e241f66527e8c5dad4d4d1200e75e"
server_key_iv: "c13086e9fd838c8c91d75c2680f34016"
server_key_version: "encryptionkey@0"
I20260923 17:24:40.128610 13367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:40.146334 13367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:40.149518 13367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:40.149715 13367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:40.151222 13367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:24:40.151273 13367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:40.151314 13367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:24:40.151351 13367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:24:40 dist-test-slave-ptl9 krb5kdc[13250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184280, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:24:40.162496 13367 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.1:46831
I20260923 17:24:40.162593 13497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.1:46831 every 8 connection(s)
I20260923 17:24:40.163506 13367 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
I20260923 17:24:40.165875 13344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:40.152276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:45737 (local address 127.11.229.62:40149)
0923 17:24:40.152436 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:40.152441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:40.153435 (+ 994us) server_negotiation.cc:407] Connection header received
0923 17:24:40.154866 (+ 1431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:40.154869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:40.154939 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:40.155039 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:40.156757 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.157495 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.158677 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.158944 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.159062 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:40.162040 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:40.162045 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:40.162049 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:40.162080 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:40.163737 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:40.164367 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:40.164371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:40.164373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:40.164467 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:40.164781 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:40.164784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:40.164785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:40.165012 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:40.165093 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:40.165643 (+ 550us) server_negotiation.cc:299] Negotiation successful
0923 17:24:40.165750 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:24:40.166652 13391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:40.152703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:40149 (local address 127.11.229.1:45737)
0923 17:24:40.153230 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:40.153316 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:40.154608 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:40.155259 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:40.155267 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:40.155796 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:40.156568 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:40.156593 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.157630 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.157633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:40.158523 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:40.158532 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.159117 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.159121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:40.159345 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:40.160001 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:40.160017 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:40.161818 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:40.163851 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:40.163857 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:40.163871 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:40.164243 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:40.164561 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:40.164563 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:40.164564 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:40.164691 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:40.165129 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:40.165133 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:40.165413 (+ 280us) client_negotiation.cc:769] Sending connection context
0923 17:24:40.165573 (+ 160us) client_negotiation.cc:240] Negotiation successful
0923 17:24:40.165815 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":134,"threads_started":1}
I20260923 17:24:40.168109 13498 heartbeater.cc:347] Connected to a master server at 127.11.229.62:40149
I20260923 17:24:40.168277 13498 heartbeater.cc:464] Registering TS with master...
I20260923 17:24:40.168813 13498 heartbeater.cc:511] Master 127.11.229.62:40149 requested a full tablet report, sending...
I20260923 17:24:40.169034 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 13367
I20260923 17:24:40.169147 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance
I20260923 17:24:40.169471 12180 external_mini_cluster.cc:1428] Setting key b1640e354c7854a677876767382acd74
I20260923 17:24:40.170593 13295 ts_manager.cc:194] Registered new tserver with Master: 73dcd776a58041f0966e5c879ffc0281 (127.11.229.1:46831)
I20260923 17:24:40.171988 13295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.1@KRBTEST.COM'} at 127.11.229.1:45737
I20260923 17:24:40.178628 13504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:40.170232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:40149 (local address 127.11.229.1:56473)
0923 17:24:40.170529 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:40.170547 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:40.170663 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:40.171025 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:40.171028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:40.171235 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:40.171351 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:40.171359 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.172532 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.172536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:40.173420 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:40.173431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.174106 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.174110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:40.174199 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:40.174725 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:40.174741 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:40.175170 (+ 429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:40.177152 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:40.177155 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:40.177156 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:40.177410 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:40.177700 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:40.177702 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:40.177703 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:40.177777 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:40.178230 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:40.178232 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:40.178301 (+ 69us) client_negotiation.cc:769] Sending connection context
0923 17:24:40.178377 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 17:24:40.178475 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":108,"threads_started":1}
I20260923 17:24:40.178817 13344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:40.170268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:56473 (local address 127.11.229.62:40149)
0923 17:24:40.170389 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:40.170392 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:40.170621 (+ 229us) server_negotiation.cc:407] Connection header received
0923 17:24:40.170788 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:40.170791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:40.170836 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:40.170917 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:40.171489 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.172370 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.173547 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.173962 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.174085 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:40.175316 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:40.175321 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:40.175325 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:40.175352 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:40.177036 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:40.177507 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:40.177512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:40.177514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:40.177602 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:40.177871 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:40.177875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:40.177878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:40.178129 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:40.178244 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:40.178475 (+ 231us) server_negotiation.cc:299] Negotiation successful
0923 17:24:40.178607 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":43}
WARNING: no policy specified for kudu/127.11.229.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab.
Entry for principal kudu/127.11.229.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab.
I20260923 17:24:40.197095 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.2:0
--local_ip_for_outbound_sockets=127.11.229.2
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:40149
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:39605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:40.311334 13507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:40.311596 13507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:40.311642 13507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:40.315659 13507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:40.315718 13507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:40.315757 13507 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:24:40.315790 13507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.2
I20260923 17:24:40.321098 13507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:39605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.229.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.229.62:40149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.2
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:40.322193 13507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:40.323196 13507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:40.332955 13512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:40.332955 13513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:40.333003 13515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:40.333179 13507 server_base.cc:1034] running on GCE node
I20260923 17:24:40.333701 13507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:40.334342 13507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:40.335533 13507 hybrid_clock.cc:648] HybridClock initialized: now 1790184280335516 us; error 29 us; skew 500 ppm
Sep 23 17:24:40 dist-test-slave-ptl9 krb5kdc[13250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184280, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:40.338446 13507 init.cc:377] Logged in from keytab as kudu/127.11.229.2@KRBTEST.COM (short username kudu)
I20260923 17:24:40.339334 13507 webserver.cc:460] Webserver started at http://127.11.229.2:35093/ using document root <none> and password file <none>
I20260923 17:24:40.339913 13507 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:40.340003 13507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:40.340241 13507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:40.342329 13507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-1/data/instance:
uuid: "b15a3f41ff394a849876108bb0c3215a"
format_stamp: "Formatted at 2026-09-23 17:24:40 on dist-test-slave-ptl9"
server_key: "a111f78cf054cd8f8405219c8ae31d48"
server_key_iv: "60b9b2a7f7217468883820c5e1498bd6"
server_key_version: "encryptionkey@0"
I20260923 17:24:40.342895 13507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance:
uuid: "b15a3f41ff394a849876108bb0c3215a"
format_stamp: "Formatted at 2026-09-23 17:24:40 on dist-test-slave-ptl9"
server_key: "a111f78cf054cd8f8405219c8ae31d48"
server_key_iv: "60b9b2a7f7217468883820c5e1498bd6"
server_key_version: "encryptionkey@0"
I20260923 17:24:40.346825 13507 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260923 17:24:40.349334 13522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:40.350551 13507 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 17:24:40.350688 13507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-1/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-1/wal
uuid: "b15a3f41ff394a849876108bb0c3215a"
format_stamp: "Formatted at 2026-09-23 17:24:40 on dist-test-slave-ptl9"
server_key: "a111f78cf054cd8f8405219c8ae31d48"
server_key_iv: "60b9b2a7f7217468883820c5e1498bd6"
server_key_version: "encryptionkey@0"
I20260923 17:24:40.350802 13507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:40.364609 13507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:40.368000 13507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:40.368243 13507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:40.369849 13507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:24:40.369930 13507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:40.370007 13507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:24:40.370048 13507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:24:40 dist-test-slave-ptl9 krb5kdc[13250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184280, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:24:40.380944 13507 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.2:33049
I20260923 17:24:40.381109 13637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.2:33049 every 8 connection(s)
I20260923 17:24:40.382175 13507 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
I20260923 17:24:40.383294 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 13507
I20260923 17:24:40.383435 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance
I20260923 17:24:40.383718 12180 external_mini_cluster.cc:1428] Setting key 8b3bdda6da7ee7a5ae2f0bb6a0c93762
I20260923 17:24:40.384594 13344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:40.370668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:56799 (local address 127.11.229.62:40149)
0923 17:24:40.370882 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:40.370887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:40.371760 (+ 873us) server_negotiation.cc:407] Connection header received
0923 17:24:40.372853 (+ 1093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:40.372859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:40.372956 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:40.373024 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:40.374859 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.375723 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.377493 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.377863 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.377994 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:40.380535 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:40.380539 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:40.380542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:40.380571 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:40.382342 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:40.382935 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:40.382938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:40.382940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:40.383038 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:40.383423 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:40.383431 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:40.383432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:40.383665 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:40.383781 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:40.384279 (+ 498us) server_negotiation.cc:299] Negotiation successful
0923 17:24:40.384409 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260923 17:24:40.385443 13531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:40.371116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:40149 (local address 127.11.229.2:56799)
0923 17:24:40.371586 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:40.371637 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:40.372592 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:40.373539 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:40.373554 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:40.374004 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:40.374710 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:40.374732 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.375843 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.375849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:40.377371 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:40.377388 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.378033 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.378038 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:40.378226 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:40.378858 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:40.378877 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:40.380360 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:40.382452 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:40.382456 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:40.382467 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:40.382811 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:40.383140 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:40.383145 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:40.383147 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:40.383290 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:40.383772 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:40.383778 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:40.384020 (+ 242us) client_negotiation.cc:769] Sending connection context
0923 17:24:40.384235 (+ 215us) client_negotiation.cc:240] Negotiation successful
0923 17:24:40.384545 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":130,"threads_started":1}
I20260923 17:24:40.386785 13638 heartbeater.cc:347] Connected to a master server at 127.11.229.62:40149
I20260923 17:24:40.386921 13638 heartbeater.cc:464] Registering TS with master...
I20260923 17:24:40.387462 13638 heartbeater.cc:511] Master 127.11.229.62:40149 requested a full tablet report, sending...
I20260923 17:24:40.388540 13294 ts_manager.cc:194] Registered new tserver with Master: b15a3f41ff394a849876108bb0c3215a (127.11.229.2:33049)
I20260923 17:24:40.389318 13294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.2@KRBTEST.COM'} at 127.11.229.2:56799
WARNING: no policy specified for kudu/127.11.229.3@KRBTEST.COM; defaulting to no policy
I20260923 17:24:40.397223 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:40.389169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:40149 (local address 127.11.229.2:52421)
0923 17:24:40.389588 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:40.389606 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:40.389740 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:40.390017 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:40.390020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:40.390243 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:40.390357 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:40.390364 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.391285 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.391288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:40.392294 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:40.392309 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.392819 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.392822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:40.392934 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:40.393501 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:40.393512 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:40.393834 (+ 322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:40.395520 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:40.395526 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:40.395528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:40.395778 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:40.396121 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:40.396124 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:40.396125 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:40.396200 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:40.396726 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:40.396728 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:40.396801 (+ 73us) client_negotiation.cc:769] Sending connection context
0923 17:24:40.396879 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 17:24:40.397072 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":116,"threads_started":1}
I20260923 17:24:40.397332 13344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:40.389282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:52421 (local address 127.11.229.62:40149)
0923 17:24:40.389419 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:40.389426 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:40.389672 (+ 246us) server_negotiation.cc:407] Connection header received
0923 17:24:40.389850 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:40.389854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:40.389899 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:40.389973 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:40.390496 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.391160 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.392411 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.392702 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.392813 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:40.393975 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:40.393979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:40.393982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:40.394012 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:40.395404 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:40.395906 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:40.395911 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:40.395912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:40.396009 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:40.396299 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:40.396304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:40.396307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:40.396630 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:40.396734 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:40.396980 (+ 246us) server_negotiation.cc:299] Negotiation successful
0923 17:24:40.397106 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab.
Entry for principal kudu/127.11.229.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab.
I20260923 17:24:40.416709 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.3:0
--local_ip_for_outbound_sockets=127.11.229.3
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:40149
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:39605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:40.535801 13647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:40.536056 13647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:40.536139 13647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:40.540158 13647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:40.540211 13647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:40.540249 13647 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:24:40.540349 13647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.3
I20260923 17:24:40.545905 13647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:39605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.229.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.229.62:40149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.3
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:40.546993 13647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:40.548041 13647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:40.557967 13652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:40.557986 13653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:40.557967 13655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:40.558200 13647 server_base.cc:1034] running on GCE node
I20260923 17:24:40.558657 13647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:40.559348 13647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:40.560515 13647 hybrid_clock.cc:648] HybridClock initialized: now 1790184280560500 us; error 28 us; skew 500 ppm
Sep 23 17:24:40 dist-test-slave-ptl9 krb5kdc[13250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184280, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:40.563390 13647 init.cc:377] Logged in from keytab as kudu/127.11.229.3@KRBTEST.COM (short username kudu)
I20260923 17:24:40.564370 13647 webserver.cc:460] Webserver started at http://127.11.229.3:39259/ using document root <none> and password file <none>
I20260923 17:24:40.564972 13647 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:40.565033 13647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:40.565304 13647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:40.567361 13647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-2/data/instance:
uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf"
format_stamp: "Formatted at 2026-09-23 17:24:40 on dist-test-slave-ptl9"
server_key: "44dce30372cfce2a2137c9d56164c62f"
server_key_iv: "c0476e99f16ea0b05c33eb45053957b7"
server_key_version: "encryptionkey@0"
I20260923 17:24:40.568001 13647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance:
uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf"
format_stamp: "Formatted at 2026-09-23 17:24:40 on dist-test-slave-ptl9"
server_key: "44dce30372cfce2a2137c9d56164c62f"
server_key_iv: "c0476e99f16ea0b05c33eb45053957b7"
server_key_version: "encryptionkey@0"
I20260923 17:24:40.571777 13647 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260923 17:24:40.574229 13662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:40.575376 13647 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:24:40.575511 13647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-2/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-2/wal
uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf"
format_stamp: "Formatted at 2026-09-23 17:24:40 on dist-test-slave-ptl9"
server_key: "44dce30372cfce2a2137c9d56164c62f"
server_key_iv: "c0476e99f16ea0b05c33eb45053957b7"
server_key_version: "encryptionkey@0"
I20260923 17:24:40.575634 13647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:40.590008 13647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:40.593448 13647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:40.593649 13647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:40.595152 13647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:24:40.595225 13647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:40.595323 13647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:24:40.595371 13647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:24:40 dist-test-slave-ptl9 krb5kdc[13250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184280, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:24:40.606460 13647 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.3:41597
I20260923 17:24:40.606544 13777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.3:41597 every 8 connection(s)
I20260923 17:24:40.607453 13647 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
I20260923 17:24:40.610666 13344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:40.595967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:46805 (local address 127.11.229.62:40149)
0923 17:24:40.596104 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:40.596108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:40.597784 (+ 1676us) server_negotiation.cc:407] Connection header received
0923 17:24:40.599056 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:40.599060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:40.599122 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:40.599225 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:40.600936 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.601870 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.603743 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.603961 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.604077 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:40.607085 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:40.607088 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:40.607090 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:40.607115 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:40.608707 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:40.609235 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:40.609239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:40.609241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:40.609370 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:40.609695 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:40.609697 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:40.609698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:40.609881 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:40.609958 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:40.610432 (+ 474us) server_negotiation.cc:299] Negotiation successful
0923 17:24:40.610534 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 17:24:40.611521 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:40.596994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:40149 (local address 127.11.229.3:46805)
0923 17:24:40.597600 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:40.597646 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:40.598827 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:40.599470 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:40.599484 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:40.600005 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:40.600777 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:40.600802 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.602036 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.602040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:40.603600 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:40.603615 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.604140 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.604147 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:40.604391 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:40.605097 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:40.605124 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:40.606894 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:40.608808 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:40.608813 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:40.608824 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:40.609124 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:40.609474 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:40.609476 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:40.609477 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:40.609608 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:40.609970 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:40.609975 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:40.610221 (+ 246us) client_negotiation.cc:769] Sending connection context
0923 17:24:40.610375 (+ 154us) client_negotiation.cc:240] Negotiation successful
0923 17:24:40.610621 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":160,"threads_started":1}
I20260923 17:24:40.612771 13778 heartbeater.cc:347] Connected to a master server at 127.11.229.62:40149
I20260923 17:24:40.612919 13778 heartbeater.cc:464] Registering TS with master...
I20260923 17:24:40.613178 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 13647
I20260923 17:24:40.613330 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance
I20260923 17:24:40.613513 13778 heartbeater.cc:511] Master 127.11.229.62:40149 requested a full tablet report, sending...
I20260923 17:24:40.613646 12180 external_mini_cluster.cc:1428] Setting key 6ef6c92958e5e4000b1de3ff4b4eec05
I20260923 17:24:40.614711 13294 ts_manager.cc:194] Registered new tserver with Master: b5bde194f59e4ea2a8cbc47dcace2cdf (127.11.229.3:41597)
I20260923 17:24:40.615459 13294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.3@KRBTEST.COM'} at 127.11.229.3:46805
I20260923 17:24:40.616014 12180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:24:40.623224 13784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:40.615188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:40149 (local address 127.11.229.3:53743)
0923 17:24:40.615485 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:40.615501 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:40.615590 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:40.616055 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:40.616059 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:40.616280 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:40.616401 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:40.616409 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.617468 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.617471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:40.618047 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:40.618055 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.618665 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.618667 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:40.618742 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:40.619292 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:40.619304 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:40.619611 (+ 307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:40.621603 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:40.621609 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:40.621611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:40.621883 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:40.622167 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:40.622173 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:40.622178 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:40.622289 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:40.622653 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:40.622659 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:40.622777 (+ 118us) client_negotiation.cc:769] Sending connection context
0923 17:24:40.622905 (+ 128us) client_negotiation.cc:240] Negotiation successful
0923 17:24:40.623074 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":100,"threads_started":1}
I20260923 17:24:40.623486 13344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:40.615728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:53743 (local address 127.11.229.62:40149)
0923 17:24:40.615831 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:40.615837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:40.615851 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:24:40.615886 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:40.615889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:40.615930 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:40.616008 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:40.616532 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.617350 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.618163 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.618537 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.618667 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:40.619756 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:40.619763 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:40.619770 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:40.619802 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:40.621482 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:40.621991 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:40.621994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:40.621995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:40.622071 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:40.622380 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:40.622382 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:40.622383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:40.622560 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:40.622633 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:40.622993 (+ 360us) server_negotiation.cc:299] Negotiation successful
0923 17:24:40.623140 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
Sep 23 17:24:40 dist-test-slave-ptl9 krb5kdc[13250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184280, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 17:24:40 dist-test-slave-ptl9 krb5kdc[13250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184280, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:24:40.642812 13344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:40.634910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34712 (local address 127.11.229.62:40149)
0923 17:24:40.635096 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:40.635100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:40.635114 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:24:40.635157 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:40.635159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:40.635194 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:40.635284 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:40.636000 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.636539 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.637402 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.637685 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.637773 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:40.640168 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:40.640171 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:40.640173 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:40.640193 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:40.641435 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:40.641822 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:40.641825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:40.641826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:40.641897 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:40.642124 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:40.642126 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:40.642127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:40.642290 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:40.642389 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:40.642583 (+ 194us) server_negotiation.cc:299] Negotiation successful
0923 17:24:40.642684 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260923 17:24:40.645828 13294 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34712:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:24:40.648845 13294 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:24:40.664637 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:40.658464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:41597 (local address 127.0.0.1:33166)
0923 17:24:40.659355 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:40.659368 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:40.659438 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:40.660160 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:40.660163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:40.660179 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:24:40.660252 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:40.660261 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.661992 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.661998 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:40.663050 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:40.663061 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.664119 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.664122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:40.664193 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:40.664210 (+ 17us) client_negotiation.cc:769] Sending connection context
0923 17:24:40.664277 (+ 67us) client_negotiation.cc:240] Negotiation successful
0923 17:24:40.664446 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"mutex_wait_us":12,"thread_start_us":88,"threads_started":1}
I20260923 17:24:40.665060 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:40.658478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40068 (local address 127.11.229.1:46831)
0923 17:24:40.659167 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:40.659183 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:40.659208 (+ 25us) server_negotiation.cc:407] Connection header received
0923 17:24:40.659437 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:40.659443 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:40.659677 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:40.659836 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:24:40.660319 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.661565 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.663151 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.663849 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.663974 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:40.664160 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:40.664744 (+ 584us) server_negotiation.cc:299] Negotiation successful
0923 17:24:40.664897 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":97,"threads_started":1}
I20260923 17:24:40.665519 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:40.659228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33166 (local address 127.11.229.3:41597)
0923 17:24:40.659711 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:40.659722 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:40.659749 (+ 27us) server_negotiation.cc:407] Connection header received
0923 17:24:40.659828 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:40.659833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:40.660013 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:40.660421 (+ 408us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:24:40.660570 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.661876 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.663193 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.663980 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.665028 (+ 1048us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:40.665208 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:40.665255 (+ 47us) server_negotiation.cc:299] Negotiation successful
0923 17:24:40.665374 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":107,"threads_started":1}
I20260923 17:24:40.664637 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:40.658323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:46831 (local address 127.0.0.1:40068)
0923 17:24:40.659001 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:40.659092 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:40.659228 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:40.660005 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:40.660011 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:40.660035 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:24:40.660208 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:40.660215 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.661710 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.661713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:40.663010 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:40.663021 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.663987 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.663991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:40.664111 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:40.664156 (+ 45us) client_negotiation.cc:769] Sending connection context
0923 17:24:40.664269 (+ 113us) client_negotiation.cc:240] Negotiation successful
0923 17:24:40.664447 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":82,"thread_start_us":119,"threads_started":1}
I20260923 17:24:40.665769 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:40.659740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:33049 (local address 127.0.0.1:55512)
0923 17:24:40.660291 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:40.660306 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:40.660424 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:40.660904 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:40.660908 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:40.660930 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:24:40.661039 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:40.661046 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.662549 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.662553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:40.663631 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:40.663641 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.665110 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.665117 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:40.665237 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:40.665296 (+ 59us) client_negotiation.cc:769] Sending connection context
0923 17:24:40.665380 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 17:24:40.665456 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":94,"threads_started":1}
I20260923 17:24:40.665844 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:40.659909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55512 (local address 127.11.229.2:33049)
0923 17:24:40.660212 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:40.660229 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:40.660417 (+ 188us) server_negotiation.cc:407] Connection header received
0923 17:24:40.660554 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:40.660559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:40.660749 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:40.660901 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:24:40.661186 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.662392 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.663782 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.664727 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.664854 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:40.665032 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:40.665572 (+ 540us) server_negotiation.cc:299] Negotiation successful
0923 17:24:40.665686 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":96,"threads_started":1}
I20260923 17:24:40.668519 13712 tablet_service.cc:1467] Processing CreateTablet for tablet 73ac46fffa1f46888f916ecb7957e47f (DEFAULT_TABLE table=test-table [id=7c83de03d041471699f1e3450acb7971]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:24:40.668519 13572 tablet_service.cc:1467] Processing CreateTablet for tablet 73ac46fffa1f46888f916ecb7957e47f (DEFAULT_TABLE table=test-table [id=7c83de03d041471699f1e3450acb7971]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:24:40.668505 13432 tablet_service.cc:1467] Processing CreateTablet for tablet 73ac46fffa1f46888f916ecb7957e47f (DEFAULT_TABLE table=test-table [id=7c83de03d041471699f1e3450acb7971]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:24:40.669690 13432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73ac46fffa1f46888f916ecb7957e47f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:40.669699 13712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73ac46fffa1f46888f916ecb7957e47f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:40.669695 13572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73ac46fffa1f46888f916ecb7957e47f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:40.676357 13801 tablet_bootstrap.cc:492] T 73ac46fffa1f46888f916ecb7957e47f P 73dcd776a58041f0966e5c879ffc0281: Bootstrap starting.
I20260923 17:24:40.677989 13802 tablet_bootstrap.cc:492] T 73ac46fffa1f46888f916ecb7957e47f P b5bde194f59e4ea2a8cbc47dcace2cdf: Bootstrap starting.
I20260923 17:24:40.678112 13803 tablet_bootstrap.cc:492] T 73ac46fffa1f46888f916ecb7957e47f P b15a3f41ff394a849876108bb0c3215a: Bootstrap starting.
I20260923 17:24:40.679723 13801 tablet_bootstrap.cc:654] T 73ac46fffa1f46888f916ecb7957e47f P 73dcd776a58041f0966e5c879ffc0281: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:40.680266 13802 tablet_bootstrap.cc:654] T 73ac46fffa1f46888f916ecb7957e47f P b5bde194f59e4ea2a8cbc47dcace2cdf: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:40.680266 13803 tablet_bootstrap.cc:654] T 73ac46fffa1f46888f916ecb7957e47f P b15a3f41ff394a849876108bb0c3215a: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:40.680718 13801 log.cc:824] T 73ac46fffa1f46888f916ecb7957e47f P 73dcd776a58041f0966e5c879ffc0281: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:40.681118 13803 log.cc:824] T 73ac46fffa1f46888f916ecb7957e47f P b15a3f41ff394a849876108bb0c3215a: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:40.681118 13802 log.cc:824] T 73ac46fffa1f46888f916ecb7957e47f P b5bde194f59e4ea2a8cbc47dcace2cdf: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:40.682957 13801 tablet_bootstrap.cc:492] T 73ac46fffa1f46888f916ecb7957e47f P 73dcd776a58041f0966e5c879ffc0281: No bootstrap required, opened a new log
I20260923 17:24:40.683035 13802 tablet_bootstrap.cc:492] T 73ac46fffa1f46888f916ecb7957e47f P b5bde194f59e4ea2a8cbc47dcace2cdf: No bootstrap required, opened a new log
I20260923 17:24:40.683184 13801 ts_tablet_manager.cc:1397] T 73ac46fffa1f46888f916ecb7957e47f P 73dcd776a58041f0966e5c879ffc0281: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260923 17:24:40.683226 13802 ts_tablet_manager.cc:1397] T 73ac46fffa1f46888f916ecb7957e47f P b5bde194f59e4ea2a8cbc47dcace2cdf: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260923 17:24:40.683470 13803 tablet_bootstrap.cc:492] T 73ac46fffa1f46888f916ecb7957e47f P b15a3f41ff394a849876108bb0c3215a: No bootstrap required, opened a new log
I20260923 17:24:40.683693 13803 ts_tablet_manager.cc:1397] T 73ac46fffa1f46888f916ecb7957e47f P b15a3f41ff394a849876108bb0c3215a: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260923 17:24:40.687260 13802 raft_consensus.cc:348] T 73ac46fffa1f46888f916ecb7957e47f P b5bde194f59e4ea2a8cbc47dcace2cdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } }
I20260923 17:24:40.687260 13801 raft_consensus.cc:348] T 73ac46fffa1f46888f916ecb7957e47f P 73dcd776a58041f0966e5c879ffc0281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } }
I20260923 17:24:40.687340 13803 raft_consensus.cc:348] T 73ac46fffa1f46888f916ecb7957e47f P b15a3f41ff394a849876108bb0c3215a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } }
I20260923 17:24:40.687553 13801 raft_consensus.cc:374] T 73ac46fffa1f46888f916ecb7957e47f P 73dcd776a58041f0966e5c879ffc0281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:40.687580 13802 raft_consensus.cc:374] T 73ac46fffa1f46888f916ecb7957e47f P b5bde194f59e4ea2a8cbc47dcace2cdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:40.687623 13801 raft_consensus.cc:729] T 73ac46fffa1f46888f916ecb7957e47f P 73dcd776a58041f0966e5c879ffc0281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73dcd776a58041f0966e5c879ffc0281, State: Initialized, Role: FOLLOWER
I20260923 17:24:40.687628 13802 raft_consensus.cc:729] T 73ac46fffa1f46888f916ecb7957e47f P b5bde194f59e4ea2a8cbc47dcace2cdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5bde194f59e4ea2a8cbc47dcace2cdf, State: Initialized, Role: FOLLOWER
I20260923 17:24:40.687659 13803 raft_consensus.cc:374] T 73ac46fffa1f46888f916ecb7957e47f P b15a3f41ff394a849876108bb0c3215a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:40.687736 13803 raft_consensus.cc:729] T 73ac46fffa1f46888f916ecb7957e47f P b15a3f41ff394a849876108bb0c3215a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b15a3f41ff394a849876108bb0c3215a, State: Initialized, Role: FOLLOWER
I20260923 17:24:40.688126 13802 consensus_queue.cc:260] T 73ac46fffa1f46888f916ecb7957e47f P b5bde194f59e4ea2a8cbc47dcace2cdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } }
I20260923 17:24:40.688098 13801 consensus_queue.cc:260] T 73ac46fffa1f46888f916ecb7957e47f P 73dcd776a58041f0966e5c879ffc0281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } }
I20260923 17:24:40.688249 13803 consensus_queue.cc:260] T 73ac46fffa1f46888f916ecb7957e47f P b15a3f41ff394a849876108bb0c3215a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } }
I20260923 17:24:40.688793 13498 heartbeater.cc:503] Master 127.11.229.62:40149 was elected leader, sending a full tablet report...
I20260923 17:24:40.689138 13801 ts_tablet_manager.cc:1428] T 73ac46fffa1f46888f916ecb7957e47f P 73dcd776a58041f0966e5c879ffc0281: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260923 17:24:40.689333 13803 ts_tablet_manager.cc:1428] T 73ac46fffa1f46888f916ecb7957e47f P b15a3f41ff394a849876108bb0c3215a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260923 17:24:40.689708 13638 heartbeater.cc:503] Master 127.11.229.62:40149 was elected leader, sending a full tablet report...
I20260923 17:24:40.690285 13802 ts_tablet_manager.cc:1428] T 73ac46fffa1f46888f916ecb7957e47f P b5bde194f59e4ea2a8cbc47dcace2cdf: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260923 17:24:40.690770 13778 heartbeater.cc:503] Master 127.11.229.62:40149 was elected leader, sending a full tablet report...
I20260923 17:24:40.789575 13807 raft_consensus.cc:482] T 73ac46fffa1f46888f916ecb7957e47f P 73dcd776a58041f0966e5c879ffc0281 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:24:40.789824 13807 raft_consensus.cc:504] T 73ac46fffa1f46888f916ecb7957e47f P 73dcd776a58041f0966e5c879ffc0281 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } }
I20260923 17:24:40.791141 13807 leader_election.cc:288] T 73ac46fffa1f46888f916ecb7957e47f P 73dcd776a58041f0966e5c879ffc0281 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5bde194f59e4ea2a8cbc47dcace2cdf (127.11.229.3:41597), b15a3f41ff394a849876108bb0c3215a (127.11.229.2:33049)
I20260923 17:24:40.795408 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:40.791522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:58929 (local address 127.11.229.2:33049)
0923 17:24:40.791653 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:40.791658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:40.792015 (+ 357us) server_negotiation.cc:407] Connection header received
0923 17:24:40.792200 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:40.792202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:40.792272 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:40.792350 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:24:40.792656 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.793187 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.794441 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.794806 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.794934 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:40.794987 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:40.795189 (+ 202us) server_negotiation.cc:299] Negotiation successful
0923 17:24:40.795269 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 17:24:40.795430 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:40.791523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:44497 (local address 127.11.229.3:41597)
0923 17:24:40.791657 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:40.791662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:40.791925 (+ 263us) server_negotiation.cc:407] Connection header received
0923 17:24:40.792110 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:40.792116 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:40.792177 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:40.792246 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:24:40.792585 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.793154 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.794416 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.794807 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.794937 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:40.794987 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:40.795249 (+ 262us) server_negotiation.cc:299] Negotiation successful
0923 17:24:40.795327 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 17:24:40.795452 13811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:40.791371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:41597 (local address 127.11.229.1:44497)
0923 17:24:40.791818 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:40.791835 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:40.791978 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:40.792285 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:40.792289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:40.792311 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:24:40.792452 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:40.792459 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.793306 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.793313 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:40.794276 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:40.794285 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.794931 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.794935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:40.795100 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:40.795119 (+ 19us) client_negotiation.cc:769] Sending connection context
0923 17:24:40.795213 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 17:24:40.795277 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":80,"thread_start_us":118,"threads_started":1}
I20260923 17:24:40.795428 13810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:40.791437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:33049 (local address 127.11.229.1:58929)
0923 17:24:40.791912 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:40.791925 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:40.792090 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:40.792382 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:40.792384 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:40.792400 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:24:40.792544 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:40.792551 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.793342 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.793346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:40.794286 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:40.794296 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.794929 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.794933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:40.795057 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:40.795085 (+ 28us) client_negotiation.cc:769] Sending connection context
0923 17:24:40.795145 (+ 60us) client_negotiation.cc:240] Negotiation successful
0923 17:24:40.795228 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":102,"thread_start_us":128,"threads_started":1}
I20260923 17:24:40.796171 13732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ac46fffa1f46888f916ecb7957e47f" candidate_uuid: "73dcd776a58041f0966e5c879ffc0281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" is_pre_election: true
I20260923 17:24:40.796254 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ac46fffa1f46888f916ecb7957e47f" candidate_uuid: "73dcd776a58041f0966e5c879ffc0281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b15a3f41ff394a849876108bb0c3215a" is_pre_election: true
I20260923 17:24:40.796525 13732 raft_consensus.cc:2454] T 73ac46fffa1f46888f916ecb7957e47f P b5bde194f59e4ea2a8cbc47dcace2cdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73dcd776a58041f0966e5c879ffc0281 in term 0.
I20260923 17:24:40.796525 13592 raft_consensus.cc:2454] T 73ac46fffa1f46888f916ecb7957e47f P b15a3f41ff394a849876108bb0c3215a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73dcd776a58041f0966e5c879ffc0281 in term 0.
I20260923 17:24:40.797035 13383 leader_election.cc:302] T 73ac46fffa1f46888f916ecb7957e47f P 73dcd776a58041f0966e5c879ffc0281 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73dcd776a58041f0966e5c879ffc0281, b15a3f41ff394a849876108bb0c3215a; no voters:
I20260923 17:24:40.797348 13807 raft_consensus.cc:2781] T 73ac46fffa1f46888f916ecb7957e47f P 73dcd776a58041f0966e5c879ffc0281 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:24:40.797435 13807 raft_consensus.cc:482] T 73ac46fffa1f46888f916ecb7957e47f P 73dcd776a58041f0966e5c879ffc0281 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:24:40.797466 13807 raft_consensus.cc:3037] T 73ac46fffa1f46888f916ecb7957e47f P 73dcd776a58041f0966e5c879ffc0281 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:40.798586 13807 raft_consensus.cc:504] T 73ac46fffa1f46888f916ecb7957e47f P 73dcd776a58041f0966e5c879ffc0281 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } }
I20260923 17:24:40.798995 13807 leader_election.cc:288] T 73ac46fffa1f46888f916ecb7957e47f P 73dcd776a58041f0966e5c879ffc0281 [CANDIDATE]: Term 1 election: Requested vote from peers b5bde194f59e4ea2a8cbc47dcace2cdf (127.11.229.3:41597), b15a3f41ff394a849876108bb0c3215a (127.11.229.2:33049)
I20260923 17:24:40.799347 13732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ac46fffa1f46888f916ecb7957e47f" candidate_uuid: "73dcd776a58041f0966e5c879ffc0281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf"
I20260923 17:24:40.799413 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ac46fffa1f46888f916ecb7957e47f" candidate_uuid: "73dcd776a58041f0966e5c879ffc0281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b15a3f41ff394a849876108bb0c3215a"
I20260923 17:24:40.799464 13732 raft_consensus.cc:3037] T 73ac46fffa1f46888f916ecb7957e47f P b5bde194f59e4ea2a8cbc47dcace2cdf [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:40.799589 13592 raft_consensus.cc:3037] T 73ac46fffa1f46888f916ecb7957e47f P b15a3f41ff394a849876108bb0c3215a [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:40.800760 13732 raft_consensus.cc:2454] T 73ac46fffa1f46888f916ecb7957e47f P b5bde194f59e4ea2a8cbc47dcace2cdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73dcd776a58041f0966e5c879ffc0281 in term 1.
I20260923 17:24:40.801082 13384 leader_election.cc:302] T 73ac46fffa1f46888f916ecb7957e47f P 73dcd776a58041f0966e5c879ffc0281 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73dcd776a58041f0966e5c879ffc0281, b5bde194f59e4ea2a8cbc47dcace2cdf; no voters:
I20260923 17:24:40.801254 13807 raft_consensus.cc:2781] T 73ac46fffa1f46888f916ecb7957e47f P 73dcd776a58041f0966e5c879ffc0281 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:24:40.801299 13592 raft_consensus.cc:2454] T 73ac46fffa1f46888f916ecb7957e47f P b15a3f41ff394a849876108bb0c3215a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73dcd776a58041f0966e5c879ffc0281 in term 1.
I20260923 17:24:40.801519 13807 raft_consensus.cc:686] T 73ac46fffa1f46888f916ecb7957e47f P 73dcd776a58041f0966e5c879ffc0281 [term 1 LEADER]: Becoming Leader. State: Replica: 73dcd776a58041f0966e5c879ffc0281, State: Running, Role: LEADER
I20260923 17:24:40.801899 13807 consensus_queue.cc:237] T 73ac46fffa1f46888f916ecb7957e47f P 73dcd776a58041f0966e5c879ffc0281 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } }
I20260923 17:24:40.805254 13294 catalog_manager.cc:5495] T 73ac46fffa1f46888f916ecb7957e47f P 73dcd776a58041f0966e5c879ffc0281 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73dcd776a58041f0966e5c879ffc0281 (127.11.229.1). New cstate: current_term: 1 leader_uuid: "73dcd776a58041f0966e5c879ffc0281" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } health_report { overall_health: UNKNOWN } } }
I20260923 17:24:40.826505 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:40.822610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40070 (local address 127.11.229.1:46831)
0923 17:24:40.822767 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:40.822773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:40.822788 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:24:40.823025 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:40.823028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:40.823108 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:40.823221 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:24:40.823484 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.823998 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.824925 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:40.825171 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:40.825314 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:40.825664 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:24:40.826081 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:24:40.826209 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:40.826290 (+ 81us) server_negotiation.cc:299] Negotiation successful
0923 17:24:40.826350 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 17:24:40.832417 13732 raft_consensus.cc:1261] T 73ac46fffa1f46888f916ecb7957e47f P b5bde194f59e4ea2a8cbc47dcace2cdf [term 1 FOLLOWER]: Refusing update from remote peer 73dcd776a58041f0966e5c879ffc0281: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:24:40.832417 13592 raft_consensus.cc:1261] T 73ac46fffa1f46888f916ecb7957e47f P b15a3f41ff394a849876108bb0c3215a [term 1 FOLLOWER]: Refusing update from remote peer 73dcd776a58041f0966e5c879ffc0281: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:24:40.833154 13807 consensus_queue.cc:1035] T 73ac46fffa1f46888f916ecb7957e47f P 73dcd776a58041f0966e5c879ffc0281 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:24:40.833391 13812 consensus_queue.cc:1035] T 73ac46fffa1f46888f916ecb7957e47f P 73dcd776a58041f0966e5c879ffc0281 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:24:40.841907 13817 mvcc.cc:205] Tried to move back new op lower bound from 7332594814282588160 to 7332594814167412736. Current Snapshot: MvccSnapshot[applied={T|T < 7332594814282588160}]
I20260923 17:24:40.852571 13292 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:24:40.852764 13292 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:24:40.855415 13292 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c5f394982af4a14b71ea8ab5e55c65f: Sending DeleteTablet for 3 replicas of tablet 73ac46fffa1f46888f916ecb7957e47f
I20260923 17:24:40.856423 13572 tablet_service.cc:1514] Processing DeleteTablet for tablet 73ac46fffa1f46888f916ecb7957e47f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:24:40 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:55512
I20260923 17:24:40.856557 13432 tablet_service.cc:1514] Processing DeleteTablet for tablet 73ac46fffa1f46888f916ecb7957e47f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:24:40 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:40068
I20260923 17:24:40.857039 13825 tablet_replica.cc:330] T 73ac46fffa1f46888f916ecb7957e47f P 73dcd776a58041f0966e5c879ffc0281: stopping tablet replica
I20260923 17:24:40.857050 13712 tablet_service.cc:1514] Processing DeleteTablet for tablet 73ac46fffa1f46888f916ecb7957e47f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:24:40 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:33166
I20260923 17:24:40.857210 13292 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34712:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260923 17:24:40.857626 13826 tablet_replica.cc:330] T 73ac46fffa1f46888f916ecb7957e47f P b5bde194f59e4ea2a8cbc47dcace2cdf: stopping tablet replica
W20260923 17:24:40.857738 13292 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:24:40.857972 13825 raft_consensus.cc:2229] T 73ac46fffa1f46888f916ecb7957e47f P 73dcd776a58041f0966e5c879ffc0281 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:24:40.857972 13826 raft_consensus.cc:2229] T 73ac46fffa1f46888f916ecb7957e47f P b5bde194f59e4ea2a8cbc47dcace2cdf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:24:40.858310 13826 raft_consensus.cc:2258] T 73ac46fffa1f46888f916ecb7957e47f P b5bde194f59e4ea2a8cbc47dcace2cdf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:24:40.858623 13824 tablet_replica.cc:330] T 73ac46fffa1f46888f916ecb7957e47f P b15a3f41ff394a849876108bb0c3215a: stopping tablet replica
I20260923 17:24:40.859594 13825 raft_consensus.cc:2258] T 73ac46fffa1f46888f916ecb7957e47f P 73dcd776a58041f0966e5c879ffc0281 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:24:40.859473 13824 raft_consensus.cc:2229] T 73ac46fffa1f46888f916ecb7957e47f P b15a3f41ff394a849876108bb0c3215a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:24:40.861145 13824 raft_consensus.cc:2258] T 73ac46fffa1f46888f916ecb7957e47f P b15a3f41ff394a849876108bb0c3215a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:24:40.861173 13825 ts_tablet_manager.cc:1905] T 73ac46fffa1f46888f916ecb7957e47f P 73dcd776a58041f0966e5c879ffc0281: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:24:40.863387 13824 ts_tablet_manager.cc:1905] T 73ac46fffa1f46888f916ecb7957e47f P b15a3f41ff394a849876108bb0c3215a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:24:40.863547 13432 tablet_service.cc:1467] Processing CreateTablet for tablet c8b584098247400b82cbfc3d422517f4 (DEFAULT_TABLE table=test-table [id=d2ff84511583474da613a56e8a798fc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:24:40.863837 13432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8b584098247400b82cbfc3d422517f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:40.864650 13825 ts_tablet_manager.cc:1918] T 73ac46fffa1f46888f916ecb7957e47f P 73dcd776a58041f0966e5c879ffc0281: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:24:40.865550 13572 tablet_service.cc:1467] Processing CreateTablet for tablet c8b584098247400b82cbfc3d422517f4 (DEFAULT_TABLE table=test-table [id=d2ff84511583474da613a56e8a798fc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:24:40.865870 13572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8b584098247400b82cbfc3d422517f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:40.866911 13824 ts_tablet_manager.cc:1918] T 73ac46fffa1f46888f916ecb7957e47f P b15a3f41ff394a849876108bb0c3215a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:24:40.866995 13824 log.cc:1196] T 73ac46fffa1f46888f916ecb7957e47f P b15a3f41ff394a849876108bb0c3215a: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-1/wal/wals/73ac46fffa1f46888f916ecb7957e47f
I20260923 17:24:40.867311 13824 ts_tablet_manager.cc:1939] T 73ac46fffa1f46888f916ecb7957e47f P b15a3f41ff394a849876108bb0c3215a: Deleting consensus metadata
I20260923 17:24:40.868332 13712 tablet_service.cc:1467] Processing CreateTablet for tablet c8b584098247400b82cbfc3d422517f4 (DEFAULT_TABLE table=test-table [id=d2ff84511583474da613a56e8a798fc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:24:40.868595 13712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8b584098247400b82cbfc3d422517f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:40.869427 13803 tablet_bootstrap.cc:492] T c8b584098247400b82cbfc3d422517f4 P b15a3f41ff394a849876108bb0c3215a: Bootstrap starting.
I20260923 17:24:40.870385 13803 tablet_bootstrap.cc:654] T c8b584098247400b82cbfc3d422517f4 P b15a3f41ff394a849876108bb0c3215a: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:40.870965 13802 tablet_bootstrap.cc:492] T c8b584098247400b82cbfc3d422517f4 P b5bde194f59e4ea2a8cbc47dcace2cdf: Bootstrap starting.
I20260923 17:24:40.871903 13279 catalog_manager.cc:4841] TS b15a3f41ff394a849876108bb0c3215a (127.11.229.2:33049): tablet 73ac46fffa1f46888f916ecb7957e47f (table test-table [id=7c83de03d041471699f1e3450acb7971]) successfully deleted
I20260923 17:24:40.871984 13802 tablet_bootstrap.cc:654] T c8b584098247400b82cbfc3d422517f4 P b5bde194f59e4ea2a8cbc47dcace2cdf: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:40.873629 13803 tablet_bootstrap.cc:492] T c8b584098247400b82cbfc3d422517f4 P b15a3f41ff394a849876108bb0c3215a: No bootstrap required, opened a new log
I20260923 17:24:40.873685 13802 tablet_bootstrap.cc:492] T c8b584098247400b82cbfc3d422517f4 P b5bde194f59e4ea2a8cbc47dcace2cdf: No bootstrap required, opened a new log
I20260923 17:24:40.873716 13803 ts_tablet_manager.cc:1397] T c8b584098247400b82cbfc3d422517f4 P b15a3f41ff394a849876108bb0c3215a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260923 17:24:40.873766 13802 ts_tablet_manager.cc:1397] T c8b584098247400b82cbfc3d422517f4 P b5bde194f59e4ea2a8cbc47dcace2cdf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 17:24:40.874176 13803 raft_consensus.cc:348] T c8b584098247400b82cbfc3d422517f4 P b15a3f41ff394a849876108bb0c3215a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } }
I20260923 17:24:40.874291 13803 raft_consensus.cc:374] T c8b584098247400b82cbfc3d422517f4 P b15a3f41ff394a849876108bb0c3215a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:40.874265 13802 raft_consensus.cc:348] T c8b584098247400b82cbfc3d422517f4 P b5bde194f59e4ea2a8cbc47dcace2cdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } }
I20260923 17:24:40.874328 13803 raft_consensus.cc:729] T c8b584098247400b82cbfc3d422517f4 P b15a3f41ff394a849876108bb0c3215a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b15a3f41ff394a849876108bb0c3215a, State: Initialized, Role: FOLLOWER
I20260923 17:24:40.874367 13802 raft_consensus.cc:374] T c8b584098247400b82cbfc3d422517f4 P b5bde194f59e4ea2a8cbc47dcace2cdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:40.874405 13802 raft_consensus.cc:729] T c8b584098247400b82cbfc3d422517f4 P b5bde194f59e4ea2a8cbc47dcace2cdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5bde194f59e4ea2a8cbc47dcace2cdf, State: Initialized, Role: FOLLOWER
I20260923 17:24:40.874468 13803 consensus_queue.cc:260] T c8b584098247400b82cbfc3d422517f4 P b15a3f41ff394a849876108bb0c3215a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } }
I20260923 17:24:40.874547 13802 consensus_queue.cc:260] T c8b584098247400b82cbfc3d422517f4 P b5bde194f59e4ea2a8cbc47dcace2cdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } }
I20260923 17:24:40.874756 13803 ts_tablet_manager.cc:1428] T c8b584098247400b82cbfc3d422517f4 P b15a3f41ff394a849876108bb0c3215a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260923 17:24:40.874790 13802 ts_tablet_manager.cc:1428] T c8b584098247400b82cbfc3d422517f4 P b5bde194f59e4ea2a8cbc47dcace2cdf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260923 17:24:40.872937 13801 tablet_bootstrap.cc:492] T c8b584098247400b82cbfc3d422517f4 P 73dcd776a58041f0966e5c879ffc0281: Bootstrap starting.
I20260923 17:24:40.877903 13801 tablet_bootstrap.cc:654] T c8b584098247400b82cbfc3d422517f4 P 73dcd776a58041f0966e5c879ffc0281: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:40.878378 13826 ts_tablet_manager.cc:1905] T 73ac46fffa1f46888f916ecb7957e47f P b5bde194f59e4ea2a8cbc47dcace2cdf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:24:40.878743 13825 log.cc:1196] T 73ac46fffa1f46888f916ecb7957e47f P 73dcd776a58041f0966e5c879ffc0281: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-0/wal/wals/73ac46fffa1f46888f916ecb7957e47f
I20260923 17:24:40.879289 13801 tablet_bootstrap.cc:492] T c8b584098247400b82cbfc3d422517f4 P 73dcd776a58041f0966e5c879ffc0281: No bootstrap required, opened a new log
I20260923 17:24:40.879381 13801 ts_tablet_manager.cc:1397] T c8b584098247400b82cbfc3d422517f4 P 73dcd776a58041f0966e5c879ffc0281: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260923 17:24:40.879954 13801 raft_consensus.cc:348] T c8b584098247400b82cbfc3d422517f4 P 73dcd776a58041f0966e5c879ffc0281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } }
I20260923 17:24:40.880100 13801 raft_consensus.cc:374] T c8b584098247400b82cbfc3d422517f4 P 73dcd776a58041f0966e5c879ffc0281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:40.880133 13801 raft_consensus.cc:729] T c8b584098247400b82cbfc3d422517f4 P 73dcd776a58041f0966e5c879ffc0281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73dcd776a58041f0966e5c879ffc0281, State: Initialized, Role: FOLLOWER
I20260923 17:24:40.880266 13801 consensus_queue.cc:260] T c8b584098247400b82cbfc3d422517f4 P 73dcd776a58041f0966e5c879ffc0281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } }
I20260923 17:24:40.880667 13801 ts_tablet_manager.cc:1428] T c8b584098247400b82cbfc3d422517f4 P 73dcd776a58041f0966e5c879ffc0281: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260923 17:24:40.881240 13825 ts_tablet_manager.cc:1939] T 73ac46fffa1f46888f916ecb7957e47f P 73dcd776a58041f0966e5c879ffc0281: Deleting consensus metadata
I20260923 17:24:40.882117 13826 ts_tablet_manager.cc:1918] T 73ac46fffa1f46888f916ecb7957e47f P b5bde194f59e4ea2a8cbc47dcace2cdf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:24:40.882203 13826 log.cc:1196] T 73ac46fffa1f46888f916ecb7957e47f P b5bde194f59e4ea2a8cbc47dcace2cdf: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-2/wal/wals/73ac46fffa1f46888f916ecb7957e47f
I20260923 17:24:40.882493 13826 ts_tablet_manager.cc:1939] T 73ac46fffa1f46888f916ecb7957e47f P b5bde194f59e4ea2a8cbc47dcace2cdf: Deleting consensus metadata
W20260923 17:24:40.883472 13639 tablet.cc:2369] T c8b584098247400b82cbfc3d422517f4 P b15a3f41ff394a849876108bb0c3215a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:24:40.883766 13279 catalog_manager.cc:4841] TS 73dcd776a58041f0966e5c879ffc0281 (127.11.229.1:46831): tablet 73ac46fffa1f46888f916ecb7957e47f (table test-table [id=7c83de03d041471699f1e3450acb7971]) successfully deleted
I20260923 17:24:40.883726 13280 catalog_manager.cc:4841] TS b5bde194f59e4ea2a8cbc47dcace2cdf (127.11.229.3:41597): tablet 73ac46fffa1f46888f916ecb7957e47f (table test-table [id=7c83de03d041471699f1e3450acb7971]) successfully deleted
W20260923 17:24:40.915226 13499 tablet.cc:2369] T c8b584098247400b82cbfc3d422517f4 P 73dcd776a58041f0966e5c879ffc0281: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:24:41.109378 13779 tablet.cc:2369] T c8b584098247400b82cbfc3d422517f4 P b5bde194f59e4ea2a8cbc47dcace2cdf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:24:41.144755 13809 raft_consensus.cc:482] T c8b584098247400b82cbfc3d422517f4 P b15a3f41ff394a849876108bb0c3215a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:24:41.144919 13809 raft_consensus.cc:504] T c8b584098247400b82cbfc3d422517f4 P b15a3f41ff394a849876108bb0c3215a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } }
I20260923 17:24:41.146160 13809 leader_election.cc:288] T c8b584098247400b82cbfc3d422517f4 P b15a3f41ff394a849876108bb0c3215a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73dcd776a58041f0966e5c879ffc0281 (127.11.229.1:46831), b5bde194f59e4ea2a8cbc47dcace2cdf (127.11.229.3:41597)
I20260923 17:24:41.151597 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:41.146568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:59965 (local address 127.11.229.1:46831)
0923 17:24:41.146828 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:41.146831 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:41.146852 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:24:41.147055 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:41.147058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:41.147130 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:41.147220 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:24:41.147688 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:41.148478 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:41.150287 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:41.150906 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:41.151038 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:41.151100 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:41.151378 (+ 278us) server_negotiation.cc:299] Negotiation successful
0923 17:24:41.151456 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260923 17:24:41.151603 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:41.146552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:53815 (local address 127.11.229.3:41597)
0923 17:24:41.146726 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:41.146731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:41.147004 (+ 273us) server_negotiation.cc:407] Connection header received
0923 17:24:41.147080 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:41.147083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:41.147130 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:41.147220 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:24:41.147688 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:41.148476 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:41.150287 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:41.150906 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:41.151038 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:41.151093 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:41.151378 (+ 285us) server_negotiation.cc:299] Negotiation successful
0923 17:24:41.151456 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 17:24:41.151683 13831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:41.146437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:46831 (local address 127.11.229.2:59965)
0923 17:24:41.146788 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:41.146802 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:41.146919 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:41.147246 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:41.147249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:41.147290 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:24:41.147562 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:41.147571 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:41.148601 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:41.148604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:41.150150 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:41.150162 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:41.151039 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:41.151043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:41.151195 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:41.151262 (+ 67us) client_negotiation.cc:769] Sending connection context
0923 17:24:41.151344 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 17:24:41.151496 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":259,"spinlock_wait_cycles":31872,"thread_start_us":101,"threads_started":1}
I20260923 17:24:41.152107 13830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:41.146446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:41597 (local address 127.11.229.2:53815)
0923 17:24:41.146894 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:41.146906 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:41.146992 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:41.147246 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:41.147249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:41.147291 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:24:41.147562 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:41.147571 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:41.148601 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:41.148604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:41.150153 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:41.150162 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:41.151039 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:41.151043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:41.151199 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:41.151262 (+ 63us) client_negotiation.cc:769] Sending connection context
0923 17:24:41.151344 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 17:24:41.151496 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":242,"spinlock_wait_cycles":6528,"thread_start_us":79,"threads_started":1}
I20260923 17:24:41.152282 13452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b584098247400b82cbfc3d422517f4" candidate_uuid: "b15a3f41ff394a849876108bb0c3215a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73dcd776a58041f0966e5c879ffc0281" is_pre_election: true
I20260923 17:24:41.152578 13452 raft_consensus.cc:2454] T c8b584098247400b82cbfc3d422517f4 P 73dcd776a58041f0966e5c879ffc0281 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b15a3f41ff394a849876108bb0c3215a in term 0.
I20260923 17:24:41.152715 13732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b584098247400b82cbfc3d422517f4" candidate_uuid: "b15a3f41ff394a849876108bb0c3215a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" is_pre_election: true
I20260923 17:24:41.152863 13732 raft_consensus.cc:2454] T c8b584098247400b82cbfc3d422517f4 P b5bde194f59e4ea2a8cbc47dcace2cdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b15a3f41ff394a849876108bb0c3215a in term 0.
I20260923 17:24:41.153100 13523 leader_election.cc:302] T c8b584098247400b82cbfc3d422517f4 P b15a3f41ff394a849876108bb0c3215a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73dcd776a58041f0966e5c879ffc0281, b15a3f41ff394a849876108bb0c3215a; no voters:
I20260923 17:24:41.153403 13809 raft_consensus.cc:2781] T c8b584098247400b82cbfc3d422517f4 P b15a3f41ff394a849876108bb0c3215a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:24:41.153469 13809 raft_consensus.cc:482] T c8b584098247400b82cbfc3d422517f4 P b15a3f41ff394a849876108bb0c3215a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:24:41.153497 13809 raft_consensus.cc:3037] T c8b584098247400b82cbfc3d422517f4 P b15a3f41ff394a849876108bb0c3215a [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:41.154573 13809 raft_consensus.cc:504] T c8b584098247400b82cbfc3d422517f4 P b15a3f41ff394a849876108bb0c3215a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } }
I20260923 17:24:41.154997 13809 leader_election.cc:288] T c8b584098247400b82cbfc3d422517f4 P b15a3f41ff394a849876108bb0c3215a [CANDIDATE]: Term 1 election: Requested vote from peers 73dcd776a58041f0966e5c879ffc0281 (127.11.229.1:46831), b5bde194f59e4ea2a8cbc47dcace2cdf (127.11.229.3:41597)
I20260923 17:24:41.155318 13452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b584098247400b82cbfc3d422517f4" candidate_uuid: "b15a3f41ff394a849876108bb0c3215a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73dcd776a58041f0966e5c879ffc0281"
I20260923 17:24:41.155448 13452 raft_consensus.cc:3037] T c8b584098247400b82cbfc3d422517f4 P 73dcd776a58041f0966e5c879ffc0281 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:41.155454 13732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b584098247400b82cbfc3d422517f4" candidate_uuid: "b15a3f41ff394a849876108bb0c3215a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf"
I20260923 17:24:41.155572 13732 raft_consensus.cc:3037] T c8b584098247400b82cbfc3d422517f4 P b5bde194f59e4ea2a8cbc47dcace2cdf [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:41.156471 13452 raft_consensus.cc:2454] T c8b584098247400b82cbfc3d422517f4 P 73dcd776a58041f0966e5c879ffc0281 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b15a3f41ff394a849876108bb0c3215a in term 1.
I20260923 17:24:41.156558 13732 raft_consensus.cc:2454] T c8b584098247400b82cbfc3d422517f4 P b5bde194f59e4ea2a8cbc47dcace2cdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b15a3f41ff394a849876108bb0c3215a in term 1.
I20260923 17:24:41.156765 13523 leader_election.cc:302] T c8b584098247400b82cbfc3d422517f4 P b15a3f41ff394a849876108bb0c3215a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73dcd776a58041f0966e5c879ffc0281, b15a3f41ff394a849876108bb0c3215a; no voters:
I20260923 17:24:41.156920 13809 raft_consensus.cc:2781] T c8b584098247400b82cbfc3d422517f4 P b15a3f41ff394a849876108bb0c3215a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:24:41.157124 13809 raft_consensus.cc:686] T c8b584098247400b82cbfc3d422517f4 P b15a3f41ff394a849876108bb0c3215a [term 1 LEADER]: Becoming Leader. State: Replica: b15a3f41ff394a849876108bb0c3215a, State: Running, Role: LEADER
I20260923 17:24:41.157416 13809 consensus_queue.cc:237] T c8b584098247400b82cbfc3d422517f4 P b15a3f41ff394a849876108bb0c3215a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } }
I20260923 17:24:41.159904 13295 catalog_manager.cc:5495] T c8b584098247400b82cbfc3d422517f4 P b15a3f41ff394a849876108bb0c3215a reported cstate change: term changed from 0 to 1, leader changed from <none> to b15a3f41ff394a849876108bb0c3215a (127.11.229.2). New cstate: current_term: 1 leader_uuid: "b15a3f41ff394a849876108bb0c3215a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } health_report { overall_health: UNKNOWN } } }
Sep 23 17:24:41 dist-test-slave-ptl9 krb5kdc[13250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184279, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:24:41.255522 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:41.246974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:40149 (local address 127.0.0.1:34720)
0923 17:24:41.247283 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:41.247296 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:41.247393 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:41.247657 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:41.247659 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:41.248060 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:41.248157 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:41.248163 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:41.248897 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:41.248899 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:41.249534 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:41.249552 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:41.249965 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:41.249967 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:41.250030 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:41.250573 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:41.250585 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:41.252011 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:41.253873 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:41.253882 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:41.253887 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:41.254192 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:41.254430 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:41.254432 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:41.254433 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:41.254557 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:41.254921 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:41.254927 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:41.255035 (+ 108us) client_negotiation.cc:769] Sending connection context
0923 17:24:41.255151 (+ 116us) client_negotiation.cc:240] Negotiation successful
0923 17:24:41.255340 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":92,"threads_started":1}
I20260923 17:24:41.255546 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:41.247080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34720 (local address 127.11.229.62:40149)
0923 17:24:41.247319 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:41.247327 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:41.247344 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:24:41.247486 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:41.247489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:41.247536 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:41.247612 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:41.248264 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:41.248806 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:41.249649 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:41.249868 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:41.249944 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:41.252157 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:41.252160 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:41.252162 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:41.252183 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:41.253767 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:41.254265 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:41.254268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:41.254270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:41.254348 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:41.254643 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:41.254646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:41.254647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:41.254831 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:41.254908 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:41.255209 (+ 301us) server_negotiation.cc:299] Negotiation successful
0923 17:24:41.255400 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":167,"thread_start_us":89,"threads_started":1}
I20260923 17:24:41.259523 13295 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:34720:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 17:24:41.260215 13295 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:24:41.264837 13572 tablet_service.cc:1467] Processing CreateTablet for tablet fce71d35865549eb8eaf52eb4e055a4e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0bf2fde524394907bfc2f480bcd6d25d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:24:41.264838 13432 tablet_service.cc:1467] Processing CreateTablet for tablet fce71d35865549eb8eaf52eb4e055a4e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0bf2fde524394907bfc2f480bcd6d25d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:24:41.265252 13572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fce71d35865549eb8eaf52eb4e055a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:41.265249 13432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fce71d35865549eb8eaf52eb4e055a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:41.267887 13801 tablet_bootstrap.cc:492] T fce71d35865549eb8eaf52eb4e055a4e P 73dcd776a58041f0966e5c879ffc0281: Bootstrap starting.
I20260923 17:24:41.267946 13803 tablet_bootstrap.cc:492] T fce71d35865549eb8eaf52eb4e055a4e P b15a3f41ff394a849876108bb0c3215a: Bootstrap starting.
I20260923 17:24:41.268518 13712 tablet_service.cc:1467] Processing CreateTablet for tablet fce71d35865549eb8eaf52eb4e055a4e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0bf2fde524394907bfc2f480bcd6d25d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:24:41.268812 13801 tablet_bootstrap.cc:654] T fce71d35865549eb8eaf52eb4e055a4e P 73dcd776a58041f0966e5c879ffc0281: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:41.268862 13803 tablet_bootstrap.cc:654] T fce71d35865549eb8eaf52eb4e055a4e P b15a3f41ff394a849876108bb0c3215a: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:41.268913 13712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fce71d35865549eb8eaf52eb4e055a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:41.270267 13803 tablet_bootstrap.cc:492] T fce71d35865549eb8eaf52eb4e055a4e P b15a3f41ff394a849876108bb0c3215a: No bootstrap required, opened a new log
I20260923 17:24:41.270373 13803 ts_tablet_manager.cc:1397] T fce71d35865549eb8eaf52eb4e055a4e P b15a3f41ff394a849876108bb0c3215a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260923 17:24:41.270437 13801 tablet_bootstrap.cc:492] T fce71d35865549eb8eaf52eb4e055a4e P 73dcd776a58041f0966e5c879ffc0281: No bootstrap required, opened a new log
I20260923 17:24:41.270532 13801 ts_tablet_manager.cc:1397] T fce71d35865549eb8eaf52eb4e055a4e P 73dcd776a58041f0966e5c879ffc0281: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260923 17:24:41.270876 13803 raft_consensus.cc:348] T fce71d35865549eb8eaf52eb4e055a4e P b15a3f41ff394a849876108bb0c3215a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } }
I20260923 17:24:41.270931 13801 raft_consensus.cc:348] T fce71d35865549eb8eaf52eb4e055a4e P 73dcd776a58041f0966e5c879ffc0281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } }
I20260923 17:24:41.271003 13803 raft_consensus.cc:374] T fce71d35865549eb8eaf52eb4e055a4e P b15a3f41ff394a849876108bb0c3215a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:41.271047 13803 raft_consensus.cc:729] T fce71d35865549eb8eaf52eb4e055a4e P b15a3f41ff394a849876108bb0c3215a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b15a3f41ff394a849876108bb0c3215a, State: Initialized, Role: FOLLOWER
I20260923 17:24:41.271060 13801 raft_consensus.cc:374] T fce71d35865549eb8eaf52eb4e055a4e P 73dcd776a58041f0966e5c879ffc0281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:41.271106 13801 raft_consensus.cc:729] T fce71d35865549eb8eaf52eb4e055a4e P 73dcd776a58041f0966e5c879ffc0281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73dcd776a58041f0966e5c879ffc0281, State: Initialized, Role: FOLLOWER
I20260923 17:24:41.271184 13803 consensus_queue.cc:260] T fce71d35865549eb8eaf52eb4e055a4e P b15a3f41ff394a849876108bb0c3215a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } }
I20260923 17:24:41.271245 13801 consensus_queue.cc:260] T fce71d35865549eb8eaf52eb4e055a4e P 73dcd776a58041f0966e5c879ffc0281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } }
I20260923 17:24:41.271458 13803 ts_tablet_manager.cc:1428] T fce71d35865549eb8eaf52eb4e055a4e P b15a3f41ff394a849876108bb0c3215a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 17:24:41.271432 13809 tablet_replica.cc:439] T fce71d35865549eb8eaf52eb4e055a4e P b15a3f41ff394a849876108bb0c3215a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } } }
I20260923 17:24:41.271524 13801 ts_tablet_manager.cc:1428] T fce71d35865549eb8eaf52eb4e055a4e P 73dcd776a58041f0966e5c879ffc0281: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 17:24:41.271574 13809 tablet_replica.cc:442] T fce71d35865549eb8eaf52eb4e055a4e P b15a3f41ff394a849876108bb0c3215a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:24:41.271759 13807 tablet_replica.cc:439] T fce71d35865549eb8eaf52eb4e055a4e P 73dcd776a58041f0966e5c879ffc0281: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } } }
I20260923 17:24:41.271903 13807 tablet_replica.cc:442] T fce71d35865549eb8eaf52eb4e055a4e P 73dcd776a58041f0966e5c879ffc0281: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:24:41.272954 13802 tablet_bootstrap.cc:492] T fce71d35865549eb8eaf52eb4e055a4e P b5bde194f59e4ea2a8cbc47dcace2cdf: Bootstrap starting.
I20260923 17:24:41.274125 13802 tablet_bootstrap.cc:654] T fce71d35865549eb8eaf52eb4e055a4e P b5bde194f59e4ea2a8cbc47dcace2cdf: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:41.275391 13802 tablet_bootstrap.cc:492] T fce71d35865549eb8eaf52eb4e055a4e P b5bde194f59e4ea2a8cbc47dcace2cdf: No bootstrap required, opened a new log
I20260923 17:24:41.275463 13802 ts_tablet_manager.cc:1397] T fce71d35865549eb8eaf52eb4e055a4e P b5bde194f59e4ea2a8cbc47dcace2cdf: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260923 17:24:41.275880 13802 raft_consensus.cc:348] T fce71d35865549eb8eaf52eb4e055a4e P b5bde194f59e4ea2a8cbc47dcace2cdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } }
I20260923 17:24:41.276013 13802 raft_consensus.cc:374] T fce71d35865549eb8eaf52eb4e055a4e P b5bde194f59e4ea2a8cbc47dcace2cdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:41.276058 13802 raft_consensus.cc:729] T fce71d35865549eb8eaf52eb4e055a4e P b5bde194f59e4ea2a8cbc47dcace2cdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5bde194f59e4ea2a8cbc47dcace2cdf, State: Initialized, Role: FOLLOWER
I20260923 17:24:41.276181 13802 consensus_queue.cc:260] T fce71d35865549eb8eaf52eb4e055a4e P b5bde194f59e4ea2a8cbc47dcace2cdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } }
I20260923 17:24:41.276458 13802 ts_tablet_manager.cc:1428] T fce71d35865549eb8eaf52eb4e055a4e P b5bde194f59e4ea2a8cbc47dcace2cdf: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260923 17:24:41.276687 13808 tablet_replica.cc:439] T fce71d35865549eb8eaf52eb4e055a4e P b5bde194f59e4ea2a8cbc47dcace2cdf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } } }
I20260923 17:24:41.276813 13808 tablet_replica.cc:442] T fce71d35865549eb8eaf52eb4e055a4e P b5bde194f59e4ea2a8cbc47dcace2cdf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:24:41.479979 13809 raft_consensus.cc:482] T fce71d35865549eb8eaf52eb4e055a4e P b15a3f41ff394a849876108bb0c3215a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:24:41.480181 13809 raft_consensus.cc:504] T fce71d35865549eb8eaf52eb4e055a4e P b15a3f41ff394a849876108bb0c3215a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } }
I20260923 17:24:41.480638 13809 leader_election.cc:288] T fce71d35865549eb8eaf52eb4e055a4e P b15a3f41ff394a849876108bb0c3215a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73dcd776a58041f0966e5c879ffc0281 (127.11.229.1:46831), b5bde194f59e4ea2a8cbc47dcace2cdf (127.11.229.3:41597)
I20260923 17:24:41.481222 13452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce71d35865549eb8eaf52eb4e055a4e" candidate_uuid: "b15a3f41ff394a849876108bb0c3215a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73dcd776a58041f0966e5c879ffc0281" is_pre_election: true
I20260923 17:24:41.481222 13732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce71d35865549eb8eaf52eb4e055a4e" candidate_uuid: "b15a3f41ff394a849876108bb0c3215a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" is_pre_election: true
I20260923 17:24:41.481422 13732 raft_consensus.cc:2454] T fce71d35865549eb8eaf52eb4e055a4e P b5bde194f59e4ea2a8cbc47dcace2cdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b15a3f41ff394a849876108bb0c3215a in term 0.
I20260923 17:24:41.481422 13452 raft_consensus.cc:2454] T fce71d35865549eb8eaf52eb4e055a4e P 73dcd776a58041f0966e5c879ffc0281 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b15a3f41ff394a849876108bb0c3215a in term 0.
I20260923 17:24:41.481769 13523 leader_election.cc:302] T fce71d35865549eb8eaf52eb4e055a4e P b15a3f41ff394a849876108bb0c3215a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 73dcd776a58041f0966e5c879ffc0281, b15a3f41ff394a849876108bb0c3215a, b5bde194f59e4ea2a8cbc47dcace2cdf; no voters:
I20260923 17:24:41.481953 13809 raft_consensus.cc:2781] T fce71d35865549eb8eaf52eb4e055a4e P b15a3f41ff394a849876108bb0c3215a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:24:41.482021 13809 raft_consensus.cc:482] T fce71d35865549eb8eaf52eb4e055a4e P b15a3f41ff394a849876108bb0c3215a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:24:41.482066 13809 raft_consensus.cc:3037] T fce71d35865549eb8eaf52eb4e055a4e P b15a3f41ff394a849876108bb0c3215a [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:41.483040 13809 raft_consensus.cc:504] T fce71d35865549eb8eaf52eb4e055a4e P b15a3f41ff394a849876108bb0c3215a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } }
I20260923 17:24:41.483389 13809 leader_election.cc:288] T fce71d35865549eb8eaf52eb4e055a4e P b15a3f41ff394a849876108bb0c3215a [CANDIDATE]: Term 1 election: Requested vote from peers 73dcd776a58041f0966e5c879ffc0281 (127.11.229.1:46831), b5bde194f59e4ea2a8cbc47dcace2cdf (127.11.229.3:41597)
I20260923 17:24:41.483702 13452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce71d35865549eb8eaf52eb4e055a4e" candidate_uuid: "b15a3f41ff394a849876108bb0c3215a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73dcd776a58041f0966e5c879ffc0281"
I20260923 17:24:41.483810 13452 raft_consensus.cc:3037] T fce71d35865549eb8eaf52eb4e055a4e P 73dcd776a58041f0966e5c879ffc0281 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:41.483876 13732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce71d35865549eb8eaf52eb4e055a4e" candidate_uuid: "b15a3f41ff394a849876108bb0c3215a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf"
I20260923 17:24:41.484042 13732 raft_consensus.cc:3037] T fce71d35865549eb8eaf52eb4e055a4e P b5bde194f59e4ea2a8cbc47dcace2cdf [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:41.484876 13452 raft_consensus.cc:2454] T fce71d35865549eb8eaf52eb4e055a4e P 73dcd776a58041f0966e5c879ffc0281 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b15a3f41ff394a849876108bb0c3215a in term 1.
I20260923 17:24:41.484957 13732 raft_consensus.cc:2454] T fce71d35865549eb8eaf52eb4e055a4e P b5bde194f59e4ea2a8cbc47dcace2cdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b15a3f41ff394a849876108bb0c3215a in term 1.
I20260923 17:24:41.485323 13524 leader_election.cc:302] T fce71d35865549eb8eaf52eb4e055a4e P b15a3f41ff394a849876108bb0c3215a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b15a3f41ff394a849876108bb0c3215a, b5bde194f59e4ea2a8cbc47dcace2cdf; no voters:
I20260923 17:24:41.485554 13809 raft_consensus.cc:2781] T fce71d35865549eb8eaf52eb4e055a4e P b15a3f41ff394a849876108bb0c3215a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:24:41.485628 13809 raft_consensus.cc:686] T fce71d35865549eb8eaf52eb4e055a4e P b15a3f41ff394a849876108bb0c3215a [term 1 LEADER]: Becoming Leader. State: Replica: b15a3f41ff394a849876108bb0c3215a, State: Running, Role: LEADER
I20260923 17:24:41.485742 13809 consensus_queue.cc:237] T fce71d35865549eb8eaf52eb4e055a4e P b15a3f41ff394a849876108bb0c3215a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } }
I20260923 17:24:41.486459 13832 tablet_replica.cc:439] T fce71d35865549eb8eaf52eb4e055a4e P b15a3f41ff394a849876108bb0c3215a: TxnStatusTablet state changed. Reason: New leader b15a3f41ff394a849876108bb0c3215a. Latest consensus state: current_term: 1 leader_uuid: "b15a3f41ff394a849876108bb0c3215a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } } }
I20260923 17:24:41.486589 13832 tablet_replica.cc:442] T fce71d35865549eb8eaf52eb4e055a4e P b15a3f41ff394a849876108bb0c3215a: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:24:41.486878 13845 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 17:24:41.487056 13845 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 17:24:41.487627 13292 catalog_manager.cc:5495] T fce71d35865549eb8eaf52eb4e055a4e P b15a3f41ff394a849876108bb0c3215a reported cstate change: term changed from 0 to 1, leader changed from <none> to b15a3f41ff394a849876108bb0c3215a (127.11.229.2). New cstate: current_term: 1 leader_uuid: "b15a3f41ff394a849876108bb0c3215a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } health_report { overall_health: UNKNOWN } } }
I20260923 17:24:41.511052 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:41.506748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:33049 (local address 127.0.0.1:55522)
0923 17:24:41.506962 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:41.506980 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:41.507123 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:41.507615 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:41.507618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:41.507626 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:24:41.507746 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:41.507752 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:41.508617 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:41.508623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:41.509425 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:41.509435 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:41.509892 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:41.509896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:41.509993 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:41.510038 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:24:41.510751 (+ 713us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:24:41.510767 (+ 16us) client_negotiation.cc:769] Sending connection context
0923 17:24:41.510838 (+ 71us) client_negotiation.cc:240] Negotiation successful
0923 17:24:41.510926 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260923 17:24:41.511080 13848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:41.506882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55522 (local address 127.11.229.2:33049)
0923 17:24:41.507339 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:41.507343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:41.507354 (+ 11us) server_negotiation.cc:407] Connection header received
0923 17:24:41.507416 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:41.507418 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:41.507468 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:41.507602 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:24:41.507883 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:41.508494 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:41.509542 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:41.509778 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:41.509891 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:41.510201 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:24:41.510649 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:24:41.510779 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:41.510846 (+ 67us) server_negotiation.cc:299] Negotiation successful
0923 17:24:41.510898 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":87,"threads_started":1}
I20260923 17:24:41.514392 13732 raft_consensus.cc:1261] T fce71d35865549eb8eaf52eb4e055a4e P b5bde194f59e4ea2a8cbc47dcace2cdf [term 1 FOLLOWER]: Refusing update from remote peer b15a3f41ff394a849876108bb0c3215a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:24:41.514544 13452 raft_consensus.cc:1261] T fce71d35865549eb8eaf52eb4e055a4e P 73dcd776a58041f0966e5c879ffc0281 [term 1 FOLLOWER]: Refusing update from remote peer b15a3f41ff394a849876108bb0c3215a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:24:41.514932 13832 consensus_queue.cc:1035] T fce71d35865549eb8eaf52eb4e055a4e P b15a3f41ff394a849876108bb0c3215a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:24:41.515117 13809 consensus_queue.cc:1035] T fce71d35865549eb8eaf52eb4e055a4e P b15a3f41ff394a849876108bb0c3215a [LEADER]: Connected to new peer: Peer: permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:24:41.516635 13808 tablet_replica.cc:439] T fce71d35865549eb8eaf52eb4e055a4e P b5bde194f59e4ea2a8cbc47dcace2cdf: TxnStatusTablet state changed. Reason: New leader b15a3f41ff394a849876108bb0c3215a. Latest consensus state: current_term: 1 leader_uuid: "b15a3f41ff394a849876108bb0c3215a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } } }
I20260923 17:24:41.516752 13808 tablet_replica.cc:442] T fce71d35865549eb8eaf52eb4e055a4e P b5bde194f59e4ea2a8cbc47dcace2cdf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:24:41.518329 13832 tablet_replica.cc:439] T fce71d35865549eb8eaf52eb4e055a4e P b15a3f41ff394a849876108bb0c3215a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b15a3f41ff394a849876108bb0c3215a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } } }
I20260923 17:24:41.518368 13807 tablet_replica.cc:439] T fce71d35865549eb8eaf52eb4e055a4e P 73dcd776a58041f0966e5c879ffc0281: TxnStatusTablet state changed. Reason: New leader b15a3f41ff394a849876108bb0c3215a. Latest consensus state: current_term: 1 leader_uuid: "b15a3f41ff394a849876108bb0c3215a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } } }
I20260923 17:24:41.518458 13832 tablet_replica.cc:442] T fce71d35865549eb8eaf52eb4e055a4e P b15a3f41ff394a849876108bb0c3215a: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:24:41.518478 13807 tablet_replica.cc:442] T fce71d35865549eb8eaf52eb4e055a4e P 73dcd776a58041f0966e5c879ffc0281: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:24:41.520990 13809 tablet_replica.cc:439] T fce71d35865549eb8eaf52eb4e055a4e P b15a3f41ff394a849876108bb0c3215a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b15a3f41ff394a849876108bb0c3215a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } } }
I20260923 17:24:41.521096 13809 tablet_replica.cc:442] T fce71d35865549eb8eaf52eb4e055a4e P b15a3f41ff394a849876108bb0c3215a: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:24:41.521098 13808 tablet_replica.cc:439] T fce71d35865549eb8eaf52eb4e055a4e P b5bde194f59e4ea2a8cbc47dcace2cdf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b15a3f41ff394a849876108bb0c3215a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } } }
I20260923 17:24:41.521201 13808 tablet_replica.cc:442] T fce71d35865549eb8eaf52eb4e055a4e P b5bde194f59e4ea2a8cbc47dcace2cdf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:24:41.522066 13807 tablet_replica.cc:439] T fce71d35865549eb8eaf52eb4e055a4e P 73dcd776a58041f0966e5c879ffc0281: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b15a3f41ff394a849876108bb0c3215a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 } } peers { permanent_uuid: "b15a3f41ff394a849876108bb0c3215a" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33049 } } peers { permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 } } }
I20260923 17:24:41.522176 13807 tablet_replica.cc:442] T fce71d35865549eb8eaf52eb4e055a4e P 73dcd776a58041f0966e5c879ffc0281: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:24:41.524097 13850 mvcc.cc:205] Tried to move back new op lower bound from 7332594817077587968 to 7332594816967249920. Current Snapshot: MvccSnapshot[applied={T|T < 7332594817077587968}]
I20260923 17:24:41.523481 13852 mvcc.cc:205] Tried to move back new op lower bound from 7332594817077587968 to 7332594816967249920. Current Snapshot: MvccSnapshot[applied={T|T < 7332594817077587968}]
I20260923 17:24:41.530087 13848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:41.526705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55534 (local address 127.11.229.2:33049)
0923 17:24:41.526853 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:41.526857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:41.527073 (+ 216us) server_negotiation.cc:407] Connection header received
0923 17:24:41.527228 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:41.527231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:41.527277 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:41.527364 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:24:41.527610 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:41.528147 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:41.528988 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:41.529221 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:41.529374 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:41.529628 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:24:41.529710 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:24:41.529799 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:41.529888 (+ 89us) server_negotiation.cc:299] Negotiation successful
0923 17:24:41.529938 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 17:24:41.534492 13856 consensus_queue.cc:1035] T c8b584098247400b82cbfc3d422517f4 P b15a3f41ff394a849876108bb0c3215a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5bde194f59e4ea2a8cbc47dcace2cdf" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:24:41.537925 13809 consensus_queue.cc:1035] T c8b584098247400b82cbfc3d422517f4 P b15a3f41ff394a849876108bb0c3215a [LEADER]: Connected to new peer: Peer: permanent_uuid: "73dcd776a58041f0966e5c879ffc0281" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 46831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:24:41.546319 13848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:41.542292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:34201 (local address 127.11.229.2:33049)
0923 17:24:41.542414 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:41.542420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:41.542731 (+ 311us) server_negotiation.cc:407] Connection header received
0923 17:24:41.542864 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:41.542866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:41.542906 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:41.542965 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:24:41.543336 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:41.543817 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:41.545208 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:41.545492 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:41.545596 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:41.545912 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:24:41.545976 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:24:41.546036 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:41.546145 (+ 109us) server_negotiation.cc:299] Negotiation successful
0923 17:24:41.546186 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":38}
I20260923 17:24:41.546394 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:41.542292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:33049 (local address 127.11.229.2:34201)
0923 17:24:41.542639 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:41.542655 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:41.542762 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:41.542989 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:41.542992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:41.542999 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:24:41.543235 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:41.543242 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:41.543914 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:41.543917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:41.545069 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:41.545080 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:41.545607 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:41.545611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:41.545727 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:41.545781 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:24:41.546055 (+ 274us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:24:41.546061 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 17:24:41.546128 (+ 67us) client_negotiation.cc:240] Negotiation successful
0923 17:24:41.546202 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":18,"thread_start_us":110,"threads_started":1}
I20260923 17:24:41.589754 13294 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:24:41.589933 13294 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:24:41.592300 13294 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c5f394982af4a14b71ea8ab5e55c65f: Sending DeleteTablet for 3 replicas of tablet c8b584098247400b82cbfc3d422517f4
I20260923 17:24:41.592949 13432 tablet_service.cc:1514] Processing DeleteTablet for tablet c8b584098247400b82cbfc3d422517f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:24:41 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:40068
I20260923 17:24:41.592949 13572 tablet_service.cc:1514] Processing DeleteTablet for tablet c8b584098247400b82cbfc3d422517f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:24:41 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:55512
I20260923 17:24:41.593051 13712 tablet_service.cc:1514] Processing DeleteTablet for tablet c8b584098247400b82cbfc3d422517f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:24:41 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:33166
I20260923 17:24:41.593410 13868 tablet_replica.cc:330] T c8b584098247400b82cbfc3d422517f4 P 73dcd776a58041f0966e5c879ffc0281: stopping tablet replica
I20260923 17:24:41.593592 13868 raft_consensus.cc:2229] T c8b584098247400b82cbfc3d422517f4 P 73dcd776a58041f0966e5c879ffc0281 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:24:41.593689 13870 tablet_replica.cc:330] T c8b584098247400b82cbfc3d422517f4 P b5bde194f59e4ea2a8cbc47dcace2cdf: stopping tablet replica
I20260923 17:24:41.593724 13868 raft_consensus.cc:2258] T c8b584098247400b82cbfc3d422517f4 P 73dcd776a58041f0966e5c879ffc0281 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:24:41.593806 13870 raft_consensus.cc:2229] T c8b584098247400b82cbfc3d422517f4 P b5bde194f59e4ea2a8cbc47dcace2cdf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:24:41.593930 13870 raft_consensus.cc:2258] T c8b584098247400b82cbfc3d422517f4 P b5bde194f59e4ea2a8cbc47dcace2cdf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:24:41.594494 13868 ts_tablet_manager.cc:1905] T c8b584098247400b82cbfc3d422517f4 P 73dcd776a58041f0966e5c879ffc0281: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:24:41.594566 13870 ts_tablet_manager.cc:1905] T c8b584098247400b82cbfc3d422517f4 P b5bde194f59e4ea2a8cbc47dcace2cdf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:24:41.597424 13869 tablet_replica.cc:330] T c8b584098247400b82cbfc3d422517f4 P b15a3f41ff394a849876108bb0c3215a: stopping tablet replica
I20260923 17:24:41.597640 13869 raft_consensus.cc:2229] T c8b584098247400b82cbfc3d422517f4 P b15a3f41ff394a849876108bb0c3215a [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:24:41.597786 13870 ts_tablet_manager.cc:1918] T c8b584098247400b82cbfc3d422517f4 P b5bde194f59e4ea2a8cbc47dcace2cdf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:24:41.597932 13870 log.cc:1196] T c8b584098247400b82cbfc3d422517f4 P b5bde194f59e4ea2a8cbc47dcace2cdf: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-2/wal/wals/c8b584098247400b82cbfc3d422517f4
I20260923 17:24:41.598039 13869 raft_consensus.cc:2258] T c8b584098247400b82cbfc3d422517f4 P b15a3f41ff394a849876108bb0c3215a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:24:41.598294 13870 ts_tablet_manager.cc:1939] T c8b584098247400b82cbfc3d422517f4 P b5bde194f59e4ea2a8cbc47dcace2cdf: Deleting consensus metadata
I20260923 17:24:41.598821 13869 ts_tablet_manager.cc:1905] T c8b584098247400b82cbfc3d422517f4 P b15a3f41ff394a849876108bb0c3215a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:24:41.599083 13280 catalog_manager.cc:4841] TS b5bde194f59e4ea2a8cbc47dcace2cdf (127.11.229.3:41597): tablet c8b584098247400b82cbfc3d422517f4 (table test-table [id=d2ff84511583474da613a56e8a798fc6]) successfully deleted
I20260923 17:24:41.599445 13868 ts_tablet_manager.cc:1918] T c8b584098247400b82cbfc3d422517f4 P 73dcd776a58041f0966e5c879ffc0281: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:24:41.599547 13868 log.cc:1196] T c8b584098247400b82cbfc3d422517f4 P 73dcd776a58041f0966e5c879ffc0281: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-0/wal/wals/c8b584098247400b82cbfc3d422517f4
I20260923 17:24:41.601619 13868 ts_tablet_manager.cc:1939] T c8b584098247400b82cbfc3d422517f4 P 73dcd776a58041f0966e5c879ffc0281: Deleting consensus metadata
I20260923 17:24:41.602411 13279 catalog_manager.cc:4841] TS 73dcd776a58041f0966e5c879ffc0281 (127.11.229.1:46831): tablet c8b584098247400b82cbfc3d422517f4 (table test-table [id=d2ff84511583474da613a56e8a798fc6]) successfully deleted
Sep 23 17:24:41 dist-test-slave-ptl9 krb5kdc[13250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184280, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.229.1@KRBTEST.COM
I20260923 17:24:41.603086 13869 ts_tablet_manager.cc:1918] T c8b584098247400b82cbfc3d422517f4 P b15a3f41ff394a849876108bb0c3215a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:24:41.603168 13869 log.cc:1196] T c8b584098247400b82cbfc3d422517f4 P b15a3f41ff394a849876108bb0c3215a: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184271597484-12180-0/minicluster-data/ts-1/wal/wals/c8b584098247400b82cbfc3d422517f4
I20260923 17:24:41.603477 13869 ts_tablet_manager.cc:1939] T c8b584098247400b82cbfc3d422517f4 P b15a3f41ff394a849876108bb0c3215a: Deleting consensus metadata
I20260923 17:24:41.604269 13279 catalog_manager.cc:4841] TS b15a3f41ff394a849876108bb0c3215a (127.11.229.2:33049): tablet c8b584098247400b82cbfc3d422517f4 (table test-table [id=d2ff84511583474da613a56e8a798fc6]) successfully deleted
I20260923 17:24:41.607461 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:41.595098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40072 (local address 127.11.229.1:46831)
0923 17:24:41.595212 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:41.595217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:41.595731 (+ 514us) server_negotiation.cc:407] Connection header received
0923 17:24:41.595925 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:41.595929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:41.595977 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:41.596045 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:41.597011 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:41.597767 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:41.598921 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:41.599251 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:41.599344 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:41.603450 (+ 4106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:41.603466 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:41.603488 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:41.603520 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:41.605530 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:41.605999 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:41.606004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:41.606005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:41.606095 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:41.606463 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:41.606466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:41.606468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:41.606727 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:41.606951 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:41.607142 (+ 191us) server_negotiation.cc:299] Negotiation successful
0923 17:24:41.607268 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260923 17:24:41.607914 13490 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40072
I20260923 17:24:41.617982 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:41.609684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34728 (local address 127.11.229.62:40149)
0923 17:24:41.609829 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:41.609833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:41.610174 (+ 341us) server_negotiation.cc:407] Connection header received
0923 17:24:41.610300 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:41.610303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:41.610374 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:41.610448 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:41.611244 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:41.611807 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:41.612661 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:41.612930 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:41.613049 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:41.614498 (+ 1449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:41.614502 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:41.614505 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:41.614548 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:41.616024 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:41.616456 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:41.616460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:41.616462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:41.616556 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:41.616849 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:41.616852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:41.616854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:41.617108 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:41.617252 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:41.617563 (+ 311us) server_negotiation.cc:299] Negotiation successful
0923 17:24:41.617698 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260923 17:24:41.618382 13294 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34728
I20260923 17:24:41.620146 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 13367
I20260923 17:24:41.629078 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 13507
I20260923 17:24:41.637167 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 13647
I20260923 17:24:41.643097 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 13263
2026-09-23T17:24:41Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3135 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:24:41 dist-test-slave-ptl9 krb5kdc[13877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:24:41 dist-test-slave-ptl9 krb5kdc[13877](info): set up 2 sockets
Sep 23 17:24:41 dist-test-slave-ptl9 krb5kdc[13877](info): commencing operation
krb5kdc: starting...
W20260923 17:24:42.693471 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:24:42 dist-test-slave-ptl9 krb5kdc[13877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:24:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:24:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.229.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
I20260923 17:24:42.814725 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:46305
--webserver_interface=127.11.229.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:38405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:46305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:42.929751 13890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:42.930030 13890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:42.930135 13890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:42.934132 13890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:24:42.934190 13890 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 17:24:42.934206 13890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:42.934226 13890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:42.934242 13890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:24:42.934293 13890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:24:42.939543 13890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:38405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:46305
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:46305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:42.940703 13890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:42.941816 13890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:42.950443 13895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:42.950505 13896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:42.950524 13898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:42.950798 13890 server_base.cc:1034] running on GCE node
I20260923 17:24:42.951432 13890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:42.952559 13890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:42.953738 13890 hybrid_clock.cc:648] HybridClock initialized: now 1790184282953717 us; error 30 us; skew 500 ppm
Sep 23 17:24:42 dist-test-slave-ptl9 krb5kdc[13877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184282, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:42.963682 13890 init.cc:377] Logged in from keytab as kudu/127.11.229.62@KRBTEST.COM (short username kudu)
I20260923 17:24:42.964705 13890 webserver.cc:460] Webserver started at http://127.11.229.62:43909/ using document root <none> and password file <none>
I20260923 17:24:42.965394 13890 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:42.965492 13890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:42.965746 13890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:42.967909 13890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "9a52102167344208bc65d3c5369023c1"
format_stamp: "Formatted at 2026-09-23 17:24:42 on dist-test-slave-ptl9"
server_key: "3fee478bf4324cf1196b84b0d65c695b"
server_key_iv: "77582d1cb5365ed08d8a0626e8986aae"
server_key_version: "encryptionkey@0"
I20260923 17:24:42.968495 13890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "9a52102167344208bc65d3c5369023c1"
format_stamp: "Formatted at 2026-09-23 17:24:42 on dist-test-slave-ptl9"
server_key: "3fee478bf4324cf1196b84b0d65c695b"
server_key_iv: "77582d1cb5365ed08d8a0626e8986aae"
server_key_version: "encryptionkey@0"
I20260923 17:24:42.972469 13890 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 17:24:42.975023 13905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:42.976316 13890 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:24:42.976467 13890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "9a52102167344208bc65d3c5369023c1"
format_stamp: "Formatted at 2026-09-23 17:24:42 on dist-test-slave-ptl9"
server_key: "3fee478bf4324cf1196b84b0d65c695b"
server_key_iv: "77582d1cb5365ed08d8a0626e8986aae"
server_key_version: "encryptionkey@0"
I20260923 17:24:42.976588 13890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:42.989425 13890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:42.992652 13890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:42.992865 13890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:43.000877 13890 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:46305
I20260923 17:24:43.000882 13967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:46305 every 8 connection(s)
I20260923 17:24:43.001996 13890 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:24:43.004830 13968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:43.010406 13968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a52102167344208bc65d3c5369023c1: Bootstrap starting.
I20260923 17:24:43.011514 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 13890
I20260923 17:24:43.011646 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:24:43.011909 12180 external_mini_cluster.cc:1428] Setting key 15c46da1de1866db3341ae9afc764371
I20260923 17:24:43.013065 13968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a52102167344208bc65d3c5369023c1: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:43.013919 13968 log.cc:824] T 00000000000000000000000000000000 P 9a52102167344208bc65d3c5369023c1: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:43.015878 13968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a52102167344208bc65d3c5369023c1: No bootstrap required, opened a new log
I20260923 17:24:43.019358 13968 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a52102167344208bc65d3c5369023c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a52102167344208bc65d3c5369023c1" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46305 } }
I20260923 17:24:43.019622 13968 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a52102167344208bc65d3c5369023c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:43.019707 13968 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a52102167344208bc65d3c5369023c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a52102167344208bc65d3c5369023c1, State: Initialized, Role: FOLLOWER
I20260923 17:24:43.020247 13968 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a52102167344208bc65d3c5369023c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a52102167344208bc65d3c5369023c1" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46305 } }
I20260923 17:24:43.020406 13968 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a52102167344208bc65d3c5369023c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:24:43.020480 13968 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a52102167344208bc65d3c5369023c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:24:43.020596 13968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a52102167344208bc65d3c5369023c1 [term 0 FOLLOWER]: Advancing to term 1
Sep 23 17:24:43 dist-test-slave-ptl9 krb5kdc[13877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:24:43.021575 13968 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a52102167344208bc65d3c5369023c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a52102167344208bc65d3c5369023c1" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46305 } }
I20260923 17:24:43.021920 13968 leader_election.cc:302] T 00000000000000000000000000000000 P 9a52102167344208bc65d3c5369023c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a52102167344208bc65d3c5369023c1; no voters:
I20260923 17:24:43.022277 13968 leader_election.cc:288] T 00000000000000000000000000000000 P 9a52102167344208bc65d3c5369023c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:24:43.022360 13973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a52102167344208bc65d3c5369023c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:24:43.022559 13973 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a52102167344208bc65d3c5369023c1 [term 1 LEADER]: Becoming Leader. State: Replica: 9a52102167344208bc65d3c5369023c1, State: Running, Role: LEADER
I20260923 17:24:43.022884 13973 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a52102167344208bc65d3c5369023c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a52102167344208bc65d3c5369023c1" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46305 } }
I20260923 17:24:43.023599 13968 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a52102167344208bc65d3c5369023c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:24:43.024581 13974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a52102167344208bc65d3c5369023c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a52102167344208bc65d3c5369023c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a52102167344208bc65d3c5369023c1" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46305 } } }
I20260923 17:24:43.025013 13974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a52102167344208bc65d3c5369023c1 [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:43.024896 13975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a52102167344208bc65d3c5369023c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a52102167344208bc65d3c5369023c1. Latest consensus state: current_term: 1 leader_uuid: "9a52102167344208bc65d3c5369023c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a52102167344208bc65d3c5369023c1" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46305 } } }
I20260923 17:24:43.025395 13975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a52102167344208bc65d3c5369023c1 [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:43.027686 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:43.013048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42302 (local address 127.11.229.62:46305)
0923 17:24:43.013531 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:43.013545 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:43.013581 (+ 36us) server_negotiation.cc:407] Connection header received
0923 17:24:43.014323 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:43.014353 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:43.014786 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:43.015133 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:43.015914 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.016929 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.017859 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.018360 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.018540 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:43.021337 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:43.021352 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:43.021366 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:43.021394 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:43.023503 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:43.024041 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:43.024047 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:43.024053 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:43.024197 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:43.024590 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:43.024596 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:43.024601 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:43.025181 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:43.026358 (+ 1177us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:43.026510 (+ 152us) server_negotiation.cc:299] Negotiation successful
0923 17:24:43.026764 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":174,"threads_started":1}
W20260923 17:24:43.028403 13988 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9a52102167344208bc65d3c5369023c1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:24:43.028491 13988 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:24:43.028606 13983 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:24:43.029366 13983 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:24:43.034390 13983 catalog_manager.cc:1347] Generated new cluster ID: 0d4c6fe8b84b456a92503be5db89bf42
I20260923 17:24:43.034456 13983 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:24:43.046252 13983 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:24:43.047247 13983 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:24:43.056401 13983 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a52102167344208bc65d3c5369023c1: Generated new TSK 0
I20260923 17:24:43.057163 13983 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.229.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab.
Entry for principal kudu/127.11.229.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab.
I20260923 17:24:43.092252 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.1:0
--local_ip_for_outbound_sockets=127.11.229.1
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:46305
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:38405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:43.214190 13994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:43.214455 13994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:43.214504 13994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:43.218533 13994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:43.218588 13994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:43.218628 13994 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 17:24:43.218642 13994 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:24:43.218673 13994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.1
I20260923 17:24:43.223907 13994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:38405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.229.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.229.62:46305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.1
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:43.224999 13994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:43.226064 13994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:43.235764 14002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:43.235760 13999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:43.235785 14000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:43.235993 13994 server_base.cc:1034] running on GCE node
I20260923 17:24:43.236521 13994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:43.237126 13994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:43.238288 13994 hybrid_clock.cc:648] HybridClock initialized: now 1790184283238277 us; error 33 us; skew 500 ppm
Sep 23 17:24:43 dist-test-slave-ptl9 krb5kdc[13877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184283, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:43.241025 13994 init.cc:377] Logged in from keytab as kudu/127.11.229.1@KRBTEST.COM (short username kudu)
I20260923 17:24:43.241954 13994 webserver.cc:460] Webserver started at http://127.11.229.1:34881/ using document root <none> and password file <none>
I20260923 17:24:43.242527 13994 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:43.242596 13994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:43.242837 13994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:43.244882 13994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-0/data/instance:
uuid: "696b0b38fed74ae1ba7e427b3c138be5"
format_stamp: "Formatted at 2026-09-23 17:24:43 on dist-test-slave-ptl9"
server_key: "62ff6d7ffd9b6bef09ae6dbe31774d2a"
server_key_iv: "3942f584a7612d9095ae59e3c88d5190"
server_key_version: "encryptionkey@0"
I20260923 17:24:43.245478 13994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance:
uuid: "696b0b38fed74ae1ba7e427b3c138be5"
format_stamp: "Formatted at 2026-09-23 17:24:43 on dist-test-slave-ptl9"
server_key: "62ff6d7ffd9b6bef09ae6dbe31774d2a"
server_key_iv: "3942f584a7612d9095ae59e3c88d5190"
server_key_version: "encryptionkey@0"
I20260923 17:24:43.249132 13994 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:24:43.251494 14009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:43.252741 13994 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:24:43.252873 13994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-0/wal
uuid: "696b0b38fed74ae1ba7e427b3c138be5"
format_stamp: "Formatted at 2026-09-23 17:24:43 on dist-test-slave-ptl9"
server_key: "62ff6d7ffd9b6bef09ae6dbe31774d2a"
server_key_iv: "3942f584a7612d9095ae59e3c88d5190"
server_key_version: "encryptionkey@0"
I20260923 17:24:43.252990 13994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:43.282219 13994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:43.285665 13994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:43.285872 13994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:43.287348 13994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:24:43.287393 13994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:43.287432 13994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:24:43.287492 13994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:43.299640 13994 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.1:33493
Sep 23 17:24:43 dist-test-slave-ptl9 krb5kdc[13877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184283, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:24:43.300028 14124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.1:33493 every 8 connection(s)
I20260923 17:24:43.300890 13994 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
I20260923 17:24:43.305691 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:43.288272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:36549 (local address 127.11.229.62:46305)
0923 17:24:43.288398 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:43.288403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:43.289556 (+ 1153us) server_negotiation.cc:407] Connection header received
0923 17:24:43.290891 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:43.290894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:43.290963 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:43.291022 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:43.292997 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.293886 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.296822 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.297245 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.297412 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:43.300688 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:43.300692 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:43.300695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:43.300739 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:43.302642 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:43.303253 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:43.303258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:43.303261 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:43.303364 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:43.303768 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:43.303771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:43.303773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:43.303965 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:43.304453 (+ 488us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:43.305398 (+ 945us) server_negotiation.cc:299] Negotiation successful
0923 17:24:43.305533 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 17:24:43.306138 14018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:43.288853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46305 (local address 127.11.229.1:36549)
0923 17:24:43.289411 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:43.289454 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:43.290622 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:43.291466 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:43.291478 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:43.292056 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:43.292779 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:43.292803 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.295105 (+ 2302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.295110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:43.296678 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:43.296693 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.297624 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.297629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:43.297844 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:43.298502 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:43.298519 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:43.300437 (+ 1918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:43.302822 (+ 2385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:43.302828 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:43.302842 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:43.303136 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:43.303473 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:43.303476 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:43.303478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:43.303654 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:43.304098 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:43.304104 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:43.304418 (+ 314us) client_negotiation.cc:769] Sending connection context
0923 17:24:43.304661 (+ 243us) client_negotiation.cc:240] Negotiation successful
0923 17:24:43.305073 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":134,"threads_started":1}
I20260923 17:24:43.307809 14125 heartbeater.cc:347] Connected to a master server at 127.11.229.62:46305
I20260923 17:24:43.307961 14125 heartbeater.cc:464] Registering TS with master...
I20260923 17:24:43.308570 14125 heartbeater.cc:511] Master 127.11.229.62:46305 requested a full tablet report, sending...
I20260923 17:24:43.310336 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 13994
I20260923 17:24:43.310384 13922 ts_manager.cc:194] Registered new tserver with Master: 696b0b38fed74ae1ba7e427b3c138be5 (127.11.229.1:33493)
I20260923 17:24:43.310451 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance
I20260923 17:24:43.310757 12180 external_mini_cluster.cc:1428] Setting key 48d54755d7b141c5238447941b5d6700
I20260923 17:24:43.311636 13922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.1@KRBTEST.COM'} at 127.11.229.1:36549
WARNING: no policy specified for kudu/127.11.229.2@KRBTEST.COM; defaulting to no policy
I20260923 17:24:43.319818 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:43.312267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:48893 (local address 127.11.229.62:46305)
0923 17:24:43.312385 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:43.312391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:43.312657 (+ 266us) server_negotiation.cc:407] Connection header received
0923 17:24:43.312853 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:43.312860 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:43.312933 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:43.312989 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:43.313747 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.314374 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.315198 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.315431 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.315551 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:43.316715 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:43.316719 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:43.316722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:43.316745 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:43.318199 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:43.318614 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:43.318619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:43.318621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:43.318705 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:43.318989 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:43.318992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:43.318994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:43.319176 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:43.319282 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:43.319538 (+ 256us) server_negotiation.cc:299] Negotiation successful
0923 17:24:43.319662 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 17:24:43.319825 14131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:43.310220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46305 (local address 127.11.229.1:48893)
0923 17:24:43.312573 (+ 2353us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:43.312592 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:43.312741 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:43.313159 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:43.313164 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:43.313447 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:43.313604 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:43.313616 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.314508 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.314510 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:43.315069 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:43.315076 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.315550 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.315554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:43.315646 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:43.316170 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:43.316184 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:43.316581 (+ 397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:43.318311 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:43.318315 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:43.318317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:43.318516 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:43.318802 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:43.318805 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:43.318807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:43.318892 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:43.319271 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:43.319274 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:43.319375 (+ 101us) client_negotiation.cc:769] Sending connection context
0923 17:24:43.319483 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 17:24:43.319625 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2255,"thread_start_us":93,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab.
Entry for principal kudu/127.11.229.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab.
I20260923 17:24:43.340008 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.2:0
--local_ip_for_outbound_sockets=127.11.229.2
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:46305
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:38405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:43.454659 14134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:43.454938 14134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:43.454986 14134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:43.459303 14134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:43.459373 14134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:43.459415 14134 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 17:24:43.459429 14134 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:24:43.459498 14134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.2
I20260923 17:24:43.465219 14134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:38405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.229.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.229.62:46305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.2
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:43.466413 14134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:43.467588 14134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:43.477507 14140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:43.477496 14139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:43.477492 14142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:43.477551 14134 server_base.cc:1034] running on GCE node
I20260923 17:24:43.478149 14134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:43.478850 14134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:43.480037 14134 hybrid_clock.cc:648] HybridClock initialized: now 1790184283479996 us; error 52 us; skew 500 ppm
Sep 23 17:24:43 dist-test-slave-ptl9 krb5kdc[13877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184283, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:43.482966 14134 init.cc:377] Logged in from keytab as kudu/127.11.229.2@KRBTEST.COM (short username kudu)
I20260923 17:24:43.483935 14134 webserver.cc:460] Webserver started at http://127.11.229.2:37015/ using document root <none> and password file <none>
I20260923 17:24:43.484531 14134 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:43.484627 14134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:43.484876 14134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:43.487077 14134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-1/data/instance:
uuid: "0cda85198f024d738628555ffa00ac19"
format_stamp: "Formatted at 2026-09-23 17:24:43 on dist-test-slave-ptl9"
server_key: "41ade5a8c33d3db31f2b6c1e990fd3f1"
server_key_iv: "15be1180165d85b6fc30b86b6f0d658a"
server_key_version: "encryptionkey@0"
I20260923 17:24:43.487697 14134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance:
uuid: "0cda85198f024d738628555ffa00ac19"
format_stamp: "Formatted at 2026-09-23 17:24:43 on dist-test-slave-ptl9"
server_key: "41ade5a8c33d3db31f2b6c1e990fd3f1"
server_key_iv: "15be1180165d85b6fc30b86b6f0d658a"
server_key_version: "encryptionkey@0"
I20260923 17:24:43.491652 14134 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:24:43.494277 14149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:43.495522 14134 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:24:43.495671 14134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-1/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-1/wal
uuid: "0cda85198f024d738628555ffa00ac19"
format_stamp: "Formatted at 2026-09-23 17:24:43 on dist-test-slave-ptl9"
server_key: "41ade5a8c33d3db31f2b6c1e990fd3f1"
server_key_iv: "15be1180165d85b6fc30b86b6f0d658a"
server_key_version: "encryptionkey@0"
I20260923 17:24:43.495795 14134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:43.516450 14134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:43.519927 14134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:43.520179 14134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:43.521906 14134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:24:43.521983 14134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:43.522056 14134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:24:43.522099 14134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:24:43 dist-test-slave-ptl9 krb5kdc[13877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184283, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:24:43.533942 14134 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.2:39963
I20260923 17:24:43.533967 14264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.2:39963 every 8 connection(s)
I20260923 17:24:43.534852 14134 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
I20260923 17:24:43.536973 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:43.522757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:36959 (local address 127.11.229.62:46305)
0923 17:24:43.522932 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:43.522937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:43.523810 (+ 873us) server_negotiation.cc:407] Connection header received
0923 17:24:43.524993 (+ 1183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:43.525000 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:43.525082 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:43.525150 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:43.526866 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.527543 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.528832 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.529106 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.529229 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:43.532830 (+ 3601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:43.532834 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:43.532837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:43.532866 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:43.534746 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:43.535489 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:43.535493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:43.535495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:43.535592 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:43.536022 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:43.536024 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:43.536026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:43.536216 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:43.536302 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:43.536743 (+ 441us) server_negotiation.cc:299] Negotiation successful
0923 17:24:43.536850 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 17:24:43.537035 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 14134
I20260923 17:24:43.537135 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance
I20260923 17:24:43.537458 12180 external_mini_cluster.cc:1428] Setting key 6b87cf82e917179935014634b325f9db
I20260923 17:24:43.538065 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:43.523170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46305 (local address 127.11.229.2:36959)
0923 17:24:43.523632 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:43.523684 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:43.524706 (+ 1022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:43.525638 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:43.525649 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:43.526144 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:43.526714 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:43.526738 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.527673 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.527676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:43.528704 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:43.528714 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.529314 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.529320 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:43.529493 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:43.530257 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:43.530283 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:43.532668 (+ 2385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:43.535031 (+ 2363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:43.535040 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:43.535057 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:43.535351 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:43.535735 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:43.535739 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:43.535742 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:43.535907 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:43.536299 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:43.536304 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:43.536530 (+ 226us) client_negotiation.cc:769] Sending connection context
0923 17:24:43.536689 (+ 159us) client_negotiation.cc:240] Negotiation successful
0923 17:24:43.537060 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":123,"threads_started":1}
I20260923 17:24:43.539410 14265 heartbeater.cc:347] Connected to a master server at 127.11.229.62:46305
I20260923 17:24:43.539570 14265 heartbeater.cc:464] Registering TS with master...
I20260923 17:24:43.540156 14265 heartbeater.cc:511] Master 127.11.229.62:46305 requested a full tablet report, sending...
I20260923 17:24:43.541303 13922 ts_manager.cc:194] Registered new tserver with Master: 0cda85198f024d738628555ffa00ac19 (127.11.229.2:39963)
I20260923 17:24:43.542012 13922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.2@KRBTEST.COM'} at 127.11.229.2:36959
WARNING: no policy specified for kudu/127.11.229.3@KRBTEST.COM; defaulting to no policy
I20260923 17:24:43.549423 14272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:43.541636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46305 (local address 127.11.229.2:34171)
0923 17:24:43.541921 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:43.541936 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:43.542036 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:43.542340 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:43.542343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:43.542575 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:43.542711 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:43.542717 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.543472 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.543476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:43.544133 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:43.544142 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.544603 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.544607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:43.544701 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:43.545232 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:43.545253 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:43.545701 (+ 448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:43.547770 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:43.547773 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:43.547774 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:43.547986 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:43.548353 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:43.548356 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:43.548357 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:43.548441 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:43.548824 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:43.548826 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:43.548955 (+ 129us) client_negotiation.cc:769] Sending connection context
0923 17:24:43.549083 (+ 128us) client_negotiation.cc:240] Negotiation successful
0923 17:24:43.549205 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":94,"threads_started":1}
I20260923 17:24:43.549425 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:43.541979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:34171 (local address 127.11.229.62:46305)
0923 17:24:43.542118 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:43.542123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:43.542136 (+ 13us) server_negotiation.cc:407] Connection header received
0923 17:24:43.542177 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:43.542182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:43.542236 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:43.542318 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:43.542828 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.543336 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.544257 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.544488 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.544599 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:43.545854 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:43.545859 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:43.545862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:43.545887 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:43.547640 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:43.548148 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:43.548152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:43.548154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:43.548248 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:43.548529 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:43.548531 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:43.548532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:43.548734 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:43.548874 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:43.549156 (+ 282us) server_negotiation.cc:299] Negotiation successful
0923 17:24:43.549248 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab.
Entry for principal kudu/127.11.229.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab.
I20260923 17:24:43.568282 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.3:0
--local_ip_for_outbound_sockets=127.11.229.3
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:46305
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:38405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:43.682269 14274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:43.682612 14274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:43.682719 14274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:43.686970 14274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:43.687059 14274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:43.687134 14274 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 17:24:43.687178 14274 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:24:43.687242 14274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.3
I20260923 17:24:43.692656 14274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:38405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.229.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.229.62:46305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.3
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:43.693995 14274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:43.695111 14274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:43.705230 14280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:43.705420 14274 server_base.cc:1034] running on GCE node
W20260923 17:24:43.705479 14279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:43.705204 14282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:43.706032 14274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:43.706789 14274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:43.707978 14274 hybrid_clock.cc:648] HybridClock initialized: now 1790184283707921 us; error 40 us; skew 500 ppm
Sep 23 17:24:43 dist-test-slave-ptl9 krb5kdc[13877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184283, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:43.711055 14274 init.cc:377] Logged in from keytab as kudu/127.11.229.3@KRBTEST.COM (short username kudu)
I20260923 17:24:43.711992 14274 webserver.cc:460] Webserver started at http://127.11.229.3:46203/ using document root <none> and password file <none>
I20260923 17:24:43.712601 14274 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:43.712697 14274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:43.712963 14274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:43.715343 14274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-2/data/instance:
uuid: "2af2232dfff944a9aceafbbd871bdd14"
format_stamp: "Formatted at 2026-09-23 17:24:43 on dist-test-slave-ptl9"
server_key: "a07f1d4fe8cb996671d7393830725594"
server_key_iv: "8ee70ab18a5c795699db9e15e26db5b7"
server_key_version: "encryptionkey@0"
I20260923 17:24:43.715939 14274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance:
uuid: "2af2232dfff944a9aceafbbd871bdd14"
format_stamp: "Formatted at 2026-09-23 17:24:43 on dist-test-slave-ptl9"
server_key: "a07f1d4fe8cb996671d7393830725594"
server_key_iv: "8ee70ab18a5c795699db9e15e26db5b7"
server_key_version: "encryptionkey@0"
I20260923 17:24:43.719861 14274 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:24:43.722502 14289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:43.723912 14274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260923 17:24:43.724087 14274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-2/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-2/wal
uuid: "2af2232dfff944a9aceafbbd871bdd14"
format_stamp: "Formatted at 2026-09-23 17:24:43 on dist-test-slave-ptl9"
server_key: "a07f1d4fe8cb996671d7393830725594"
server_key_iv: "8ee70ab18a5c795699db9e15e26db5b7"
server_key_version: "encryptionkey@0"
I20260923 17:24:43.724217 14274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:43.736101 14274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:43.739781 14274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:43.740028 14274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:43.741585 14274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:24:43.741670 14274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:43.741766 14274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:24:43.741816 14274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:43.752604 14274 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.3:33357
I20260923 17:24:43.752688 14404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.3:33357 every 8 connection(s)
Sep 23 17:24:43 dist-test-slave-ptl9 krb5kdc[13877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184283, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:24:43.753774 14274 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
I20260923 17:24:43.755119 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 14274
I20260923 17:24:43.755260 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance
I20260923 17:24:43.755585 12180 external_mini_cluster.cc:1428] Setting key 8a553765c2e1b34c5bfd13121a587fbe
I20260923 17:24:43.757624 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:43.742455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:34851 (local address 127.11.229.62:46305)
0923 17:24:43.742595 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:43.742599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:43.743726 (+ 1127us) server_negotiation.cc:407] Connection header received
0923 17:24:43.744861 (+ 1135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:43.744867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:43.744936 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:43.744996 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:43.747331 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.748258 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.749916 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.750315 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.750453 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:43.753411 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:43.753416 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:43.753418 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:43.753449 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:43.755126 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:43.755806 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:43.755809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:43.755810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:43.755923 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:43.756290 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:43.756292 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:43.756294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:43.756537 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:43.756620 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:43.757310 (+ 690us) server_negotiation.cc:299] Negotiation successful
0923 17:24:43.757440 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 17:24:43.758450 14298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:43.743055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46305 (local address 127.11.229.3:34851)
0923 17:24:43.743571 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:43.743614 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:43.744603 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:43.745834 (+ 1231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:43.745841 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:43.746352 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:43.747112 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:43.747129 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.748398 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.748403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:43.749768 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:43.749779 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.750505 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.750511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:43.750747 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:43.751506 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:43.751540 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:43.753178 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:43.755290 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:43.755296 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:43.755312 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:43.755689 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:43.756027 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:43.756030 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:43.756032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:43.756185 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:43.756667 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:43.756677 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:43.756979 (+ 302us) client_negotiation.cc:769] Sending connection context
0923 17:24:43.757217 (+ 238us) client_negotiation.cc:240] Negotiation successful
0923 17:24:43.757568 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":131,"threads_started":1}
I20260923 17:24:43.759559 14405 heartbeater.cc:347] Connected to a master server at 127.11.229.62:46305
I20260923 17:24:43.759692 14405 heartbeater.cc:464] Registering TS with master...
I20260923 17:24:43.760247 14405 heartbeater.cc:511] Master 127.11.229.62:46305 requested a full tablet report, sending...
I20260923 17:24:43.761386 13918 ts_manager.cc:194] Registered new tserver with Master: 2af2232dfff944a9aceafbbd871bdd14 (127.11.229.3:33357)
I20260923 17:24:43.762173 13918 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.3@KRBTEST.COM'} at 127.11.229.3:34851
I20260923 17:24:43.769479 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:43.762023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:46571 (local address 127.11.229.62:46305)
0923 17:24:43.762171 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:43.762175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:43.762660 (+ 485us) server_negotiation.cc:407] Connection header received
0923 17:24:43.762813 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:43.762815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:43.762861 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:43.762939 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:43.763353 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.763895 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.764733 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.764953 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.765064 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:43.766245 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:43.766249 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:43.766252 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:43.766300 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:43.767830 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:43.768253 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:43.768256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:43.768258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:43.768343 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:43.768624 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:43.768628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:43.768630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:43.768810 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:43.768917 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:43.769192 (+ 275us) server_negotiation.cc:299] Negotiation successful
0923 17:24:43.769330 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 17:24:43.769487 14411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:43.761947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46305 (local address 127.11.229.3:46571)
0923 17:24:43.762579 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:43.762594 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:43.762703 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:43.762964 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:43.762966 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:43.763136 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:43.763236 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:43.763242 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.764022 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.764026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:43.764600 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:43.764608 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.765068 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.765072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:43.765167 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:43.765763 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:43.765779 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:43.766100 (+ 321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:43.767941 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:43.767945 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:43.767948 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:43.768150 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:43.768435 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:43.768439 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:43.768441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:43.768529 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:43.768918 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:43.768922 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:43.769019 (+ 97us) client_negotiation.cc:769] Sending connection context
0923 17:24:43.769127 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 17:24:43.769272 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":88,"threads_started":1}
I20260923 17:24:43.770514 12180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 17:24:43 dist-test-slave-ptl9 krb5kdc[13877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 17:24:43 dist-test-slave-ptl9 krb5kdc[13877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:24:43.796622 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:43.788400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42312 (local address 127.11.229.62:46305)
0923 17:24:43.788531 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:43.788535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:43.788547 (+ 12us) server_negotiation.cc:407] Connection header received
0923 17:24:43.788592 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:43.788594 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:43.788633 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:43.788700 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:43.789529 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.790052 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.790909 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.791118 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.791204 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:43.793723 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:43.793727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:43.793730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:43.793757 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:43.795221 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:43.795610 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:43.795613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:43.795614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:43.795692 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:43.795915 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:43.795917 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:43.795918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:43.796091 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:43.796201 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:43.796393 (+ 192us) server_negotiation.cc:299] Negotiation successful
0923 17:24:43.796489 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 17:24:43.799914 13922 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42312:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:24:43.803264 13922 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:24:43.818038 14423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:43.812225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55848 (local address 127.11.229.3:33357)
0923 17:24:43.812690 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:43.812703 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:43.812731 (+ 28us) server_negotiation.cc:407] Connection header received
0923 17:24:43.812816 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:43.812824 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:43.813063 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:43.813254 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:24:43.813802 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.815043 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.816536 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.817324 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.817436 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:43.817608 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:43.817773 (+ 165us) server_negotiation.cc:299] Negotiation successful
0923 17:24:43.817885 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":113,"threads_started":1}
I20260923 17:24:43.818037 14422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:43.812081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:33357 (local address 127.0.0.1:55848)
0923 17:24:43.812468 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:43.812524 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:43.812678 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:43.813491 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:43.813499 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:43.813526 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:24:43.813662 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:43.813669 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.815192 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.815196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:43.816406 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:43.816417 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.817434 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.817438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:43.817564 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:43.817608 (+ 44us) client_negotiation.cc:769] Sending connection context
0923 17:24:43.817726 (+ 118us) client_negotiation.cc:240] Negotiation successful
0923 17:24:43.817844 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":4,"thread_start_us":80,"threads_started":1}
I20260923 17:24:43.818296 14424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:43.812849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:39963 (local address 127.0.0.1:33552)
0923 17:24:43.813237 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:43.813252 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:43.813369 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:43.814019 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:43.814022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:43.814045 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:24:43.814153 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:43.814159 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.815480 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.815483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:43.816753 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:43.816764 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.817881 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.817885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:43.817975 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:43.817998 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 17:24:43.818051 (+ 53us) client_negotiation.cc:240] Negotiation successful
0923 17:24:43.818129 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":43,"thread_start_us":103,"threads_started":1}
I20260923 17:24:43.818470 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:43.813211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:33493 (local address 127.0.0.1:32782)
0923 17:24:43.813561 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:43.813575 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:43.813652 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:43.814416 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:43.814419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:43.814437 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:24:43.814519 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:43.814528 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.815927 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.815931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:43.816999 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:43.817006 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.818072 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.818077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:43.818175 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:43.818198 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 17:24:43.818255 (+ 57us) client_negotiation.cc:240] Negotiation successful
0923 17:24:43.818308 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":117,"threads_started":1}
I20260923 17:24:43.819345 14427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:43.813658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32782 (local address 127.11.229.1:33493)
0923 17:24:43.813958 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:43.813985 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:43.814016 (+ 31us) server_negotiation.cc:407] Connection header received
0923 17:24:43.814090 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:43.814094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:43.814281 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:43.814433 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:24:43.814646 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.815787 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.817113 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.817932 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.818925 (+ 993us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:43.819068 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:43.819116 (+ 48us) server_negotiation.cc:299] Negotiation successful
0923 17:24:43.819203 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":122,"threads_started":1}
I20260923 17:24:43.819712 14425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:43.813055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33552 (local address 127.11.229.2:39963)
0923 17:24:43.813504 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:43.813515 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:43.813536 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:24:43.813608 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:43.813613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:43.813821 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:43.814018 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:24:43.814278 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.815355 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.816885 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.817743 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.818463 (+ 720us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:43.818632 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:43.818688 (+ 56us) server_negotiation.cc:299] Negotiation successful
0923 17:24:43.818783 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":116,"threads_started":1}
I20260923 17:24:43.821863 14046 tablet_service.cc:1467] Processing CreateTablet for tablet 28e444192eb1486ea5bc8c56e7bc3a0c (DEFAULT_TABLE table=test-table [id=917aa1f81730469daa7dcb9ea261284a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:24:43.821903 14339 tablet_service.cc:1467] Processing CreateTablet for tablet 28e444192eb1486ea5bc8c56e7bc3a0c (DEFAULT_TABLE table=test-table [id=917aa1f81730469daa7dcb9ea261284a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:24:43.822108 14199 tablet_service.cc:1467] Processing CreateTablet for tablet 28e444192eb1486ea5bc8c56e7bc3a0c (DEFAULT_TABLE table=test-table [id=917aa1f81730469daa7dcb9ea261284a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:24:43.822929 14046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28e444192eb1486ea5bc8c56e7bc3a0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:43.822960 14199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28e444192eb1486ea5bc8c56e7bc3a0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:43.822968 14339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28e444192eb1486ea5bc8c56e7bc3a0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:43.829406 14428 tablet_bootstrap.cc:492] T 28e444192eb1486ea5bc8c56e7bc3a0c P 696b0b38fed74ae1ba7e427b3c138be5: Bootstrap starting.
I20260923 17:24:43.831013 14429 tablet_bootstrap.cc:492] T 28e444192eb1486ea5bc8c56e7bc3a0c P 0cda85198f024d738628555ffa00ac19: Bootstrap starting.
I20260923 17:24:43.831852 14428 tablet_bootstrap.cc:654] T 28e444192eb1486ea5bc8c56e7bc3a0c P 696b0b38fed74ae1ba7e427b3c138be5: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:43.832450 14430 tablet_bootstrap.cc:492] T 28e444192eb1486ea5bc8c56e7bc3a0c P 2af2232dfff944a9aceafbbd871bdd14: Bootstrap starting.
I20260923 17:24:43.832988 14428 log.cc:824] T 28e444192eb1486ea5bc8c56e7bc3a0c P 696b0b38fed74ae1ba7e427b3c138be5: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:43.833335 14429 tablet_bootstrap.cc:654] T 28e444192eb1486ea5bc8c56e7bc3a0c P 0cda85198f024d738628555ffa00ac19: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:43.834148 14429 log.cc:824] T 28e444192eb1486ea5bc8c56e7bc3a0c P 0cda85198f024d738628555ffa00ac19: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:43.834599 14430 tablet_bootstrap.cc:654] T 28e444192eb1486ea5bc8c56e7bc3a0c P 2af2232dfff944a9aceafbbd871bdd14: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:43.835276 14428 tablet_bootstrap.cc:492] T 28e444192eb1486ea5bc8c56e7bc3a0c P 696b0b38fed74ae1ba7e427b3c138be5: No bootstrap required, opened a new log
I20260923 17:24:43.835425 14430 log.cc:824] T 28e444192eb1486ea5bc8c56e7bc3a0c P 2af2232dfff944a9aceafbbd871bdd14: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:43.835489 14428 ts_tablet_manager.cc:1397] T 28e444192eb1486ea5bc8c56e7bc3a0c P 696b0b38fed74ae1ba7e427b3c138be5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 17:24:43.836040 14429 tablet_bootstrap.cc:492] T 28e444192eb1486ea5bc8c56e7bc3a0c P 0cda85198f024d738628555ffa00ac19: No bootstrap required, opened a new log
I20260923 17:24:43.836282 14429 ts_tablet_manager.cc:1397] T 28e444192eb1486ea5bc8c56e7bc3a0c P 0cda85198f024d738628555ffa00ac19: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20260923 17:24:43.837604 14430 tablet_bootstrap.cc:492] T 28e444192eb1486ea5bc8c56e7bc3a0c P 2af2232dfff944a9aceafbbd871bdd14: No bootstrap required, opened a new log
I20260923 17:24:43.837796 14430 ts_tablet_manager.cc:1397] T 28e444192eb1486ea5bc8c56e7bc3a0c P 2af2232dfff944a9aceafbbd871bdd14: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260923 17:24:43.839684 14429 raft_consensus.cc:348] T 28e444192eb1486ea5bc8c56e7bc3a0c P 0cda85198f024d738628555ffa00ac19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } }
I20260923 17:24:43.839983 14429 raft_consensus.cc:374] T 28e444192eb1486ea5bc8c56e7bc3a0c P 0cda85198f024d738628555ffa00ac19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:43.840062 14429 raft_consensus.cc:729] T 28e444192eb1486ea5bc8c56e7bc3a0c P 0cda85198f024d738628555ffa00ac19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cda85198f024d738628555ffa00ac19, State: Initialized, Role: FOLLOWER
I20260923 17:24:43.839908 14428 raft_consensus.cc:348] T 28e444192eb1486ea5bc8c56e7bc3a0c P 696b0b38fed74ae1ba7e427b3c138be5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } }
I20260923 17:24:43.840152 14428 raft_consensus.cc:374] T 28e444192eb1486ea5bc8c56e7bc3a0c P 696b0b38fed74ae1ba7e427b3c138be5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:43.840216 14428 raft_consensus.cc:729] T 28e444192eb1486ea5bc8c56e7bc3a0c P 696b0b38fed74ae1ba7e427b3c138be5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696b0b38fed74ae1ba7e427b3c138be5, State: Initialized, Role: FOLLOWER
I20260923 17:24:43.840560 14429 consensus_queue.cc:260] T 28e444192eb1486ea5bc8c56e7bc3a0c P 0cda85198f024d738628555ffa00ac19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } }
I20260923 17:24:43.840713 14428 consensus_queue.cc:260] T 28e444192eb1486ea5bc8c56e7bc3a0c P 696b0b38fed74ae1ba7e427b3c138be5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } }
I20260923 17:24:43.841418 14125 heartbeater.cc:503] Master 127.11.229.62:46305 was elected leader, sending a full tablet report...
I20260923 17:24:43.841707 14429 ts_tablet_manager.cc:1428] T 28e444192eb1486ea5bc8c56e7bc3a0c P 0cda85198f024d738628555ffa00ac19: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260923 17:24:43.841728 14428 ts_tablet_manager.cc:1428] T 28e444192eb1486ea5bc8c56e7bc3a0c P 696b0b38fed74ae1ba7e427b3c138be5: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260923 17:24:43.841665 14430 raft_consensus.cc:348] T 28e444192eb1486ea5bc8c56e7bc3a0c P 2af2232dfff944a9aceafbbd871bdd14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } }
I20260923 17:24:43.841909 14430 raft_consensus.cc:374] T 28e444192eb1486ea5bc8c56e7bc3a0c P 2af2232dfff944a9aceafbbd871bdd14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:43.841969 14430 raft_consensus.cc:729] T 28e444192eb1486ea5bc8c56e7bc3a0c P 2af2232dfff944a9aceafbbd871bdd14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2af2232dfff944a9aceafbbd871bdd14, State: Initialized, Role: FOLLOWER
I20260923 17:24:43.842037 14265 heartbeater.cc:503] Master 127.11.229.62:46305 was elected leader, sending a full tablet report...
I20260923 17:24:43.842383 14430 consensus_queue.cc:260] T 28e444192eb1486ea5bc8c56e7bc3a0c P 2af2232dfff944a9aceafbbd871bdd14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } }
I20260923 17:24:43.843446 14405 heartbeater.cc:503] Master 127.11.229.62:46305 was elected leader, sending a full tablet report...
I20260923 17:24:43.844412 14430 ts_tablet_manager.cc:1428] T 28e444192eb1486ea5bc8c56e7bc3a0c P 2af2232dfff944a9aceafbbd871bdd14: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260923 17:24:43.867838 14434 raft_consensus.cc:482] T 28e444192eb1486ea5bc8c56e7bc3a0c P 0cda85198f024d738628555ffa00ac19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:24:43.868029 14434 raft_consensus.cc:504] T 28e444192eb1486ea5bc8c56e7bc3a0c P 0cda85198f024d738628555ffa00ac19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } }
I20260923 17:24:43.869182 14434 leader_election.cc:288] T 28e444192eb1486ea5bc8c56e7bc3a0c P 0cda85198f024d738628555ffa00ac19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2af2232dfff944a9aceafbbd871bdd14 (127.11.229.3:33357), 696b0b38fed74ae1ba7e427b3c138be5 (127.11.229.1:33493)
I20260923 17:24:43.872812 14423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:43.869564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:57197 (local address 127.11.229.3:33357)
0923 17:24:43.869690 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:43.869712 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:43.869729 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:24:43.869816 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:43.869820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:43.869873 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:43.869964 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:24:43.870250 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.870778 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.871842 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.872255 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.872385 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:43.872438 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:43.872617 (+ 179us) server_negotiation.cc:299] Negotiation successful
0923 17:24:43.872684 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 17:24:43.872812 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:43.869441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:33357 (local address 127.11.229.2:57197)
0923 17:24:43.869576 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:43.869593 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:43.869696 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:43.869982 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:43.869985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:43.870004 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:24:43.870136 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:43.870142 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.870893 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.870897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:43.871707 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:43.871716 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.872385 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.872388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:43.872496 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:43.872514 (+ 18us) client_negotiation.cc:769] Sending connection context
0923 17:24:43.872602 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 17:24:43.872679 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260923 17:24:43.873522 14359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e444192eb1486ea5bc8c56e7bc3a0c" candidate_uuid: "0cda85198f024d738628555ffa00ac19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2af2232dfff944a9aceafbbd871bdd14" is_pre_election: true
I20260923 17:24:43.873874 14359 raft_consensus.cc:2454] T 28e444192eb1486ea5bc8c56e7bc3a0c P 2af2232dfff944a9aceafbbd871bdd14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cda85198f024d738628555ffa00ac19 in term 0.
I20260923 17:24:43.873986 14427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:43.869912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:49279 (local address 127.11.229.1:33493)
0923 17:24:43.870123 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:43.870127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:43.870262 (+ 135us) server_negotiation.cc:407] Connection header received
0923 17:24:43.870415 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:43.870418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:43.870466 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:43.870550 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:24:43.871175 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.871681 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.872694 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.873376 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.873515 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:43.873564 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:43.873771 (+ 207us) server_negotiation.cc:299] Negotiation successful
0923 17:24:43.873852 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260923 17:24:43.874006 14437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:43.869755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:33493 (local address 127.11.229.2:49279)
0923 17:24:43.870159 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:43.870178 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:43.870296 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:43.870562 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:43.870565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:43.870581 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:24:43.870715 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:43.870720 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.871796 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.871801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:43.872550 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:43.872559 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.873515 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.873519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:43.873623 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:43.873652 (+ 29us) client_negotiation.cc:769] Sending connection context
0923 17:24:43.873747 (+ 95us) client_negotiation.cc:240] Negotiation successful
0923 17:24:43.873817 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":138,"threads_started":1}
I20260923 17:24:43.874446 14151 leader_election.cc:302] T 28e444192eb1486ea5bc8c56e7bc3a0c P 0cda85198f024d738628555ffa00ac19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cda85198f024d738628555ffa00ac19, 2af2232dfff944a9aceafbbd871bdd14; no voters:
I20260923 17:24:43.874616 14073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e444192eb1486ea5bc8c56e7bc3a0c" candidate_uuid: "0cda85198f024d738628555ffa00ac19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696b0b38fed74ae1ba7e427b3c138be5" is_pre_election: true
I20260923 17:24:43.874760 14434 raft_consensus.cc:2781] T 28e444192eb1486ea5bc8c56e7bc3a0c P 0cda85198f024d738628555ffa00ac19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:24:43.874852 14434 raft_consensus.cc:482] T 28e444192eb1486ea5bc8c56e7bc3a0c P 0cda85198f024d738628555ffa00ac19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:24:43.874898 14434 raft_consensus.cc:3037] T 28e444192eb1486ea5bc8c56e7bc3a0c P 0cda85198f024d738628555ffa00ac19 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:43.874945 14073 raft_consensus.cc:2454] T 28e444192eb1486ea5bc8c56e7bc3a0c P 696b0b38fed74ae1ba7e427b3c138be5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cda85198f024d738628555ffa00ac19 in term 0.
I20260923 17:24:43.876106 14434 raft_consensus.cc:504] T 28e444192eb1486ea5bc8c56e7bc3a0c P 0cda85198f024d738628555ffa00ac19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } }
I20260923 17:24:43.876458 14434 leader_election.cc:288] T 28e444192eb1486ea5bc8c56e7bc3a0c P 0cda85198f024d738628555ffa00ac19 [CANDIDATE]: Term 1 election: Requested vote from peers 2af2232dfff944a9aceafbbd871bdd14 (127.11.229.3:33357), 696b0b38fed74ae1ba7e427b3c138be5 (127.11.229.1:33493)
I20260923 17:24:43.876878 14073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e444192eb1486ea5bc8c56e7bc3a0c" candidate_uuid: "0cda85198f024d738628555ffa00ac19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696b0b38fed74ae1ba7e427b3c138be5"
I20260923 17:24:43.876869 14359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e444192eb1486ea5bc8c56e7bc3a0c" candidate_uuid: "0cda85198f024d738628555ffa00ac19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2af2232dfff944a9aceafbbd871bdd14"
I20260923 17:24:43.877003 14073 raft_consensus.cc:3037] T 28e444192eb1486ea5bc8c56e7bc3a0c P 696b0b38fed74ae1ba7e427b3c138be5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:43.877003 14359 raft_consensus.cc:3037] T 28e444192eb1486ea5bc8c56e7bc3a0c P 2af2232dfff944a9aceafbbd871bdd14 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:43.878379 14359 raft_consensus.cc:2454] T 28e444192eb1486ea5bc8c56e7bc3a0c P 2af2232dfff944a9aceafbbd871bdd14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cda85198f024d738628555ffa00ac19 in term 1.
I20260923 17:24:43.878379 14073 raft_consensus.cc:2454] T 28e444192eb1486ea5bc8c56e7bc3a0c P 696b0b38fed74ae1ba7e427b3c138be5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cda85198f024d738628555ffa00ac19 in term 1.
I20260923 17:24:43.878691 14151 leader_election.cc:302] T 28e444192eb1486ea5bc8c56e7bc3a0c P 0cda85198f024d738628555ffa00ac19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cda85198f024d738628555ffa00ac19, 696b0b38fed74ae1ba7e427b3c138be5; no voters:
I20260923 17:24:43.878857 14434 raft_consensus.cc:2781] T 28e444192eb1486ea5bc8c56e7bc3a0c P 0cda85198f024d738628555ffa00ac19 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:24:43.879102 14434 raft_consensus.cc:686] T 28e444192eb1486ea5bc8c56e7bc3a0c P 0cda85198f024d738628555ffa00ac19 [term 1 LEADER]: Becoming Leader. State: Replica: 0cda85198f024d738628555ffa00ac19, State: Running, Role: LEADER
I20260923 17:24:43.879494 14434 consensus_queue.cc:237] T 28e444192eb1486ea5bc8c56e7bc3a0c P 0cda85198f024d738628555ffa00ac19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } }
I20260923 17:24:43.883005 13921 catalog_manager.cc:5495] T 28e444192eb1486ea5bc8c56e7bc3a0c P 0cda85198f024d738628555ffa00ac19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cda85198f024d738628555ffa00ac19 (127.11.229.2). New cstate: current_term: 1 leader_uuid: "0cda85198f024d738628555ffa00ac19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } health_report { overall_health: UNKNOWN } } }
Sep 23 17:24:43 dist-test-slave-ptl9 krb5kdc[13877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184282, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:24:43.927452 14445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:43.919132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46305 (local address 127.0.0.1:42318)
0923 17:24:43.919461 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:43.919478 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:43.919601 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:43.919844 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:43.919846 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:43.920252 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:43.920353 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:43.920359 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.921076 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.921078 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:43.921667 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:43.921674 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.922114 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.922116 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:43.922180 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:43.922800 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:43.922812 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:43.924297 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:43.925976 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:43.925985 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:43.925990 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:43.926237 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:43.926477 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:43.926479 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:43.926480 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:43.926559 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:43.926909 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:43.926914 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:43.927022 (+ 108us) client_negotiation.cc:769] Sending connection context
0923 17:24:43.927118 (+ 96us) client_negotiation.cc:240] Negotiation successful
0923 17:24:43.927255 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":137,"threads_started":1}
I20260923 17:24:43.927443 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:43.919164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42318 (local address 127.11.229.62:46305)
0923 17:24:43.919288 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:43.919292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:43.919572 (+ 280us) server_negotiation.cc:407] Connection header received
0923 17:24:43.919684 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:43.919686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:43.919725 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:43.919802 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:43.920460 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.920982 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.921799 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.922008 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.922093 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:43.924426 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:43.924428 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:43.924430 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:43.924452 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:43.925876 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:43.926324 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:43.926326 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:43.926327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:43.926398 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:43.926636 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:43.926638 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:43.926639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:43.926820 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:43.926893 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:43.927179 (+ 286us) server_negotiation.cc:299] Negotiation successful
0923 17:24:43.927309 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":45}
I20260923 17:24:43.931458 13921 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:42318:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 17:24:43.932230 13921 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:24:43.937443 14199 tablet_service.cc:1467] Processing CreateTablet for tablet a7d0ec2d54174d23854520fe1431dd31 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c1b0ab9aa0f34b7ab78f0aac6ba6da35]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:24:43.937441 14046 tablet_service.cc:1467] Processing CreateTablet for tablet a7d0ec2d54174d23854520fe1431dd31 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c1b0ab9aa0f34b7ab78f0aac6ba6da35]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:24:43.937686 14339 tablet_service.cc:1467] Processing CreateTablet for tablet a7d0ec2d54174d23854520fe1431dd31 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c1b0ab9aa0f34b7ab78f0aac6ba6da35]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:24:43.938251 14199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7d0ec2d54174d23854520fe1431dd31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:43.938251 14046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7d0ec2d54174d23854520fe1431dd31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:43.938325 14339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7d0ec2d54174d23854520fe1431dd31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:43.941051 14430 tablet_bootstrap.cc:492] T a7d0ec2d54174d23854520fe1431dd31 P 2af2232dfff944a9aceafbbd871bdd14: Bootstrap starting.
I20260923 17:24:43.941278 14428 tablet_bootstrap.cc:492] T a7d0ec2d54174d23854520fe1431dd31 P 696b0b38fed74ae1ba7e427b3c138be5: Bootstrap starting.
I20260923 17:24:43.941262 14429 tablet_bootstrap.cc:492] T a7d0ec2d54174d23854520fe1431dd31 P 0cda85198f024d738628555ffa00ac19: Bootstrap starting.
I20260923 17:24:43.942179 14430 tablet_bootstrap.cc:654] T a7d0ec2d54174d23854520fe1431dd31 P 2af2232dfff944a9aceafbbd871bdd14: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:43.942215 14429 tablet_bootstrap.cc:654] T a7d0ec2d54174d23854520fe1431dd31 P 0cda85198f024d738628555ffa00ac19: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:43.942255 14428 tablet_bootstrap.cc:654] T a7d0ec2d54174d23854520fe1431dd31 P 696b0b38fed74ae1ba7e427b3c138be5: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:43.943497 14428 tablet_bootstrap.cc:492] T a7d0ec2d54174d23854520fe1431dd31 P 696b0b38fed74ae1ba7e427b3c138be5: No bootstrap required, opened a new log
I20260923 17:24:43.943586 14428 ts_tablet_manager.cc:1397] T a7d0ec2d54174d23854520fe1431dd31 P 696b0b38fed74ae1ba7e427b3c138be5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 17:24:43.943884 14429 tablet_bootstrap.cc:492] T a7d0ec2d54174d23854520fe1431dd31 P 0cda85198f024d738628555ffa00ac19: No bootstrap required, opened a new log
I20260923 17:24:43.943977 14429 ts_tablet_manager.cc:1397] T a7d0ec2d54174d23854520fe1431dd31 P 0cda85198f024d738628555ffa00ac19: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260923 17:24:43.944077 14430 tablet_bootstrap.cc:492] T a7d0ec2d54174d23854520fe1431dd31 P 2af2232dfff944a9aceafbbd871bdd14: No bootstrap required, opened a new log
I20260923 17:24:43.944188 14430 ts_tablet_manager.cc:1397] T a7d0ec2d54174d23854520fe1431dd31 P 2af2232dfff944a9aceafbbd871bdd14: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 17:24:43.944120 14428 raft_consensus.cc:348] T a7d0ec2d54174d23854520fe1431dd31 P 696b0b38fed74ae1ba7e427b3c138be5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } }
I20260923 17:24:43.944242 14428 raft_consensus.cc:374] T a7d0ec2d54174d23854520fe1431dd31 P 696b0b38fed74ae1ba7e427b3c138be5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:43.944283 14428 raft_consensus.cc:729] T a7d0ec2d54174d23854520fe1431dd31 P 696b0b38fed74ae1ba7e427b3c138be5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696b0b38fed74ae1ba7e427b3c138be5, State: Initialized, Role: FOLLOWER
I20260923 17:24:43.944414 14428 consensus_queue.cc:260] T a7d0ec2d54174d23854520fe1431dd31 P 696b0b38fed74ae1ba7e427b3c138be5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } }
I20260923 17:24:43.944468 14429 raft_consensus.cc:348] T a7d0ec2d54174d23854520fe1431dd31 P 0cda85198f024d738628555ffa00ac19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } }
I20260923 17:24:43.944581 14429 raft_consensus.cc:374] T a7d0ec2d54174d23854520fe1431dd31 P 0cda85198f024d738628555ffa00ac19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:43.944618 14429 raft_consensus.cc:729] T a7d0ec2d54174d23854520fe1431dd31 P 0cda85198f024d738628555ffa00ac19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cda85198f024d738628555ffa00ac19, State: Initialized, Role: FOLLOWER
I20260923 17:24:43.944702 14435 tablet_replica.cc:439] T a7d0ec2d54174d23854520fe1431dd31 P 696b0b38fed74ae1ba7e427b3c138be5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } } }
I20260923 17:24:43.944705 14430 raft_consensus.cc:348] T a7d0ec2d54174d23854520fe1431dd31 P 2af2232dfff944a9aceafbbd871bdd14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } }
I20260923 17:24:43.944758 14429 consensus_queue.cc:260] T a7d0ec2d54174d23854520fe1431dd31 P 0cda85198f024d738628555ffa00ac19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } }
I20260923 17:24:43.944844 14430 raft_consensus.cc:374] T a7d0ec2d54174d23854520fe1431dd31 P 2af2232dfff944a9aceafbbd871bdd14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:43.944851 14435 tablet_replica.cc:442] T a7d0ec2d54174d23854520fe1431dd31 P 696b0b38fed74ae1ba7e427b3c138be5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:24:43.944887 14430 raft_consensus.cc:729] T a7d0ec2d54174d23854520fe1431dd31 P 2af2232dfff944a9aceafbbd871bdd14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2af2232dfff944a9aceafbbd871bdd14, State: Initialized, Role: FOLLOWER
I20260923 17:24:43.945025 14429 ts_tablet_manager.cc:1428] T a7d0ec2d54174d23854520fe1431dd31 P 0cda85198f024d738628555ffa00ac19: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 17:24:43.945041 14430 consensus_queue.cc:260] T a7d0ec2d54174d23854520fe1431dd31 P 2af2232dfff944a9aceafbbd871bdd14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } }
I20260923 17:24:43.945353 14430 ts_tablet_manager.cc:1428] T a7d0ec2d54174d23854520fe1431dd31 P 2af2232dfff944a9aceafbbd871bdd14: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260923 17:24:43.945361 14436 tablet_replica.cc:439] T a7d0ec2d54174d23854520fe1431dd31 P 2af2232dfff944a9aceafbbd871bdd14: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } } }
I20260923 17:24:43.945518 14436 tablet_replica.cc:442] T a7d0ec2d54174d23854520fe1431dd31 P 2af2232dfff944a9aceafbbd871bdd14: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:24:43.945451 14434 tablet_replica.cc:439] T a7d0ec2d54174d23854520fe1431dd31 P 0cda85198f024d738628555ffa00ac19: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } } }
I20260923 17:24:43.945575 14434 tablet_replica.cc:442] T a7d0ec2d54174d23854520fe1431dd31 P 0cda85198f024d738628555ffa00ac19: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:24:43.946321 14428 ts_tablet_manager.cc:1428] T a7d0ec2d54174d23854520fe1431dd31 P 696b0b38fed74ae1ba7e427b3c138be5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260923 17:24:43.994364 14435 raft_consensus.cc:482] T a7d0ec2d54174d23854520fe1431dd31 P 696b0b38fed74ae1ba7e427b3c138be5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:24:43.994505 14435 raft_consensus.cc:504] T a7d0ec2d54174d23854520fe1431dd31 P 696b0b38fed74ae1ba7e427b3c138be5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } }
I20260923 17:24:43.995841 14435 leader_election.cc:288] T a7d0ec2d54174d23854520fe1431dd31 P 696b0b38fed74ae1ba7e427b3c138be5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cda85198f024d738628555ffa00ac19 (127.11.229.2:39963), 2af2232dfff944a9aceafbbd871bdd14 (127.11.229.3:33357)
I20260923 17:24:44.001324 14425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:43.996183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:58101 (local address 127.11.229.2:39963)
0923 17:24:43.996335 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:43.996340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:43.996521 (+ 181us) server_negotiation.cc:407] Connection header received
0923 17:24:43.996708 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:43.996712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:43.996767 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:43.996850 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:24:43.997168 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.997893 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.999978 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.000608 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.000721 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:44.000774 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:44.001109 (+ 335us) server_negotiation.cc:299] Negotiation successful
0923 17:24:44.001182 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 17:24:44.001350 14449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:43.996077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:39963 (local address 127.11.229.1:58101)
0923 17:24:43.996392 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:43.996414 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:43.996572 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:43.996870 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:43.996873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:43.996891 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:24:43.997051 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:43.997061 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.998006 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.998009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:43.999821 (+ 1812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:43.999834 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.000860 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.000863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:44.000947 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:44.000971 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 17:24:44.001068 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 17:24:44.001145 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":7,"thread_start_us":139,"threads_started":1}
I20260923 17:24:44.001387 14423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:43.996657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:53559 (local address 127.11.229.3:33357)
0923 17:24:43.996781 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:43.996787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:43.996967 (+ 180us) server_negotiation.cc:407] Connection header received
0923 17:24:43.997149 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:43.997152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:43.997201 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:43.997316 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:24:43.997619 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.998246 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.999958 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.000598 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.000721 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:44.000774 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:44.001165 (+ 391us) server_negotiation.cc:299] Negotiation successful
0923 17:24:44.001246 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 17:24:44.001900 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d0ec2d54174d23854520fe1431dd31" candidate_uuid: "696b0b38fed74ae1ba7e427b3c138be5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cda85198f024d738628555ffa00ac19" is_pre_election: true
I20260923 17:24:44.002177 14219 raft_consensus.cc:2454] T a7d0ec2d54174d23854520fe1431dd31 P 0cda85198f024d738628555ffa00ac19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696b0b38fed74ae1ba7e427b3c138be5 in term 0.
I20260923 17:24:44.002445 14450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:43.996537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:33357 (local address 127.11.229.1:53559)
0923 17:24:43.996892 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:43.996904 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:43.997006 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:43.997325 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:43.997328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:43.997347 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:24:43.997476 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:43.997486 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:43.998416 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:43.998421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:43.999824 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:43.999834 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.000822 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.000826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:44.000930 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:44.001021 (+ 91us) client_negotiation.cc:769] Sending connection context
0923 17:24:44.001580 (+ 559us) client_negotiation.cc:240] Negotiation successful
0923 17:24:44.002251 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":20,"thread_start_us":116,"threads_started":1}
I20260923 17:24:44.002673 14011 leader_election.cc:302] T a7d0ec2d54174d23854520fe1431dd31 P 696b0b38fed74ae1ba7e427b3c138be5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cda85198f024d738628555ffa00ac19, 696b0b38fed74ae1ba7e427b3c138be5; no voters:
I20260923 17:24:44.002982 14435 raft_consensus.cc:2781] T a7d0ec2d54174d23854520fe1431dd31 P 696b0b38fed74ae1ba7e427b3c138be5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:24:44.003075 14435 raft_consensus.cc:482] T a7d0ec2d54174d23854520fe1431dd31 P 696b0b38fed74ae1ba7e427b3c138be5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:24:44.003120 14435 raft_consensus.cc:3037] T a7d0ec2d54174d23854520fe1431dd31 P 696b0b38fed74ae1ba7e427b3c138be5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:44.003387 14359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d0ec2d54174d23854520fe1431dd31" candidate_uuid: "696b0b38fed74ae1ba7e427b3c138be5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2af2232dfff944a9aceafbbd871bdd14" is_pre_election: true
I20260923 17:24:44.003535 14359 raft_consensus.cc:2454] T a7d0ec2d54174d23854520fe1431dd31 P 2af2232dfff944a9aceafbbd871bdd14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696b0b38fed74ae1ba7e427b3c138be5 in term 0.
I20260923 17:24:44.004328 14435 raft_consensus.cc:504] T a7d0ec2d54174d23854520fe1431dd31 P 696b0b38fed74ae1ba7e427b3c138be5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } }
I20260923 17:24:44.004734 14435 leader_election.cc:288] T a7d0ec2d54174d23854520fe1431dd31 P 696b0b38fed74ae1ba7e427b3c138be5 [CANDIDATE]: Term 1 election: Requested vote from peers 0cda85198f024d738628555ffa00ac19 (127.11.229.2:39963), 2af2232dfff944a9aceafbbd871bdd14 (127.11.229.3:33357)
W20260923 17:24:44.005014 14406 tablet.cc:2369] T 28e444192eb1486ea5bc8c56e7bc3a0c P 2af2232dfff944a9aceafbbd871bdd14: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:24:44.005342 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d0ec2d54174d23854520fe1431dd31" candidate_uuid: "696b0b38fed74ae1ba7e427b3c138be5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cda85198f024d738628555ffa00ac19"
I20260923 17:24:44.005347 14359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d0ec2d54174d23854520fe1431dd31" candidate_uuid: "696b0b38fed74ae1ba7e427b3c138be5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2af2232dfff944a9aceafbbd871bdd14"
I20260923 17:24:44.005463 14359 raft_consensus.cc:3037] T a7d0ec2d54174d23854520fe1431dd31 P 2af2232dfff944a9aceafbbd871bdd14 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:44.005470 14219 raft_consensus.cc:3037] T a7d0ec2d54174d23854520fe1431dd31 P 0cda85198f024d738628555ffa00ac19 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:44.006631 14219 raft_consensus.cc:2454] T a7d0ec2d54174d23854520fe1431dd31 P 0cda85198f024d738628555ffa00ac19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696b0b38fed74ae1ba7e427b3c138be5 in term 1.
I20260923 17:24:44.006642 14359 raft_consensus.cc:2454] T a7d0ec2d54174d23854520fe1431dd31 P 2af2232dfff944a9aceafbbd871bdd14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696b0b38fed74ae1ba7e427b3c138be5 in term 1.
I20260923 17:24:44.006948 14011 leader_election.cc:302] T a7d0ec2d54174d23854520fe1431dd31 P 696b0b38fed74ae1ba7e427b3c138be5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cda85198f024d738628555ffa00ac19, 696b0b38fed74ae1ba7e427b3c138be5; no voters:
I20260923 17:24:44.007244 14435 raft_consensus.cc:2781] T a7d0ec2d54174d23854520fe1431dd31 P 696b0b38fed74ae1ba7e427b3c138be5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:24:44.007457 14435 raft_consensus.cc:686] T a7d0ec2d54174d23854520fe1431dd31 P 696b0b38fed74ae1ba7e427b3c138be5 [term 1 LEADER]: Becoming Leader. State: Replica: 696b0b38fed74ae1ba7e427b3c138be5, State: Running, Role: LEADER
I20260923 17:24:44.007813 14435 consensus_queue.cc:237] T a7d0ec2d54174d23854520fe1431dd31 P 696b0b38fed74ae1ba7e427b3c138be5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } }
I20260923 17:24:44.010764 14451 tablet_replica.cc:439] T a7d0ec2d54174d23854520fe1431dd31 P 696b0b38fed74ae1ba7e427b3c138be5: TxnStatusTablet state changed. Reason: New leader 696b0b38fed74ae1ba7e427b3c138be5. Latest consensus state: current_term: 1 leader_uuid: "696b0b38fed74ae1ba7e427b3c138be5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } } }
I20260923 17:24:44.010878 14451 tablet_replica.cc:442] T a7d0ec2d54174d23854520fe1431dd31 P 696b0b38fed74ae1ba7e427b3c138be5: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:24:44.011178 14453 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 17:24:44.011406 14453 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 17:24:44.011262 13921 catalog_manager.cc:5495] T a7d0ec2d54174d23854520fe1431dd31 P 696b0b38fed74ae1ba7e427b3c138be5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696b0b38fed74ae1ba7e427b3c138be5 (127.11.229.1). New cstate: current_term: 1 leader_uuid: "696b0b38fed74ae1ba7e427b3c138be5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } health_report { overall_health: UNKNOWN } } }
I20260923 17:24:44.032087 14445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:44.027933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:33493 (local address 127.0.0.1:32788)
0923 17:24:44.028139 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:44.028156 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:44.028259 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:44.028605 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:44.028607 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:44.028613 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:24:44.028722 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:44.028726 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.029771 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.029774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:44.030388 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:44.030396 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.030946 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.030949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:44.031033 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:44.031080 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:24:44.031829 (+ 749us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:24:44.031841 (+ 12us) client_negotiation.cc:769] Sending connection context
0923 17:24:44.031893 (+ 52us) client_negotiation.cc:240] Negotiation successful
0923 17:24:44.031963 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260923 17:24:44.032124 14427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:44.028092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32788 (local address 127.11.229.1:33493)
0923 17:24:44.028222 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:44.028227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:44.028244 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:24:44.028406 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:44.028409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:44.028484 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:44.028588 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:24:44.028826 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.029652 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.030495 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.030837 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.030936 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:44.031217 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:24:44.031734 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:24:44.031845 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:44.031908 (+ 63us) server_negotiation.cc:299] Negotiation successful
0923 17:24:44.031971 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260923 17:24:44.036744 14266 tablet.cc:2369] T 28e444192eb1486ea5bc8c56e7bc3a0c P 0cda85198f024d738628555ffa00ac19: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:24:44.037426 14219 raft_consensus.cc:1261] T a7d0ec2d54174d23854520fe1431dd31 P 0cda85198f024d738628555ffa00ac19 [term 1 FOLLOWER]: Refusing update from remote peer 696b0b38fed74ae1ba7e427b3c138be5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:24:44.037554 14359 raft_consensus.cc:1261] T a7d0ec2d54174d23854520fe1431dd31 P 2af2232dfff944a9aceafbbd871bdd14 [term 1 FOLLOWER]: Refusing update from remote peer 696b0b38fed74ae1ba7e427b3c138be5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:24:44.038175 14435 consensus_queue.cc:1035] T a7d0ec2d54174d23854520fe1431dd31 P 696b0b38fed74ae1ba7e427b3c138be5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:24:44.038368 14451 consensus_queue.cc:1035] T a7d0ec2d54174d23854520fe1431dd31 P 696b0b38fed74ae1ba7e427b3c138be5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:24:44.041697 14434 tablet_replica.cc:439] T a7d0ec2d54174d23854520fe1431dd31 P 0cda85198f024d738628555ffa00ac19: TxnStatusTablet state changed. Reason: New leader 696b0b38fed74ae1ba7e427b3c138be5. Latest consensus state: current_term: 1 leader_uuid: "696b0b38fed74ae1ba7e427b3c138be5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } } }
I20260923 17:24:44.041821 14434 tablet_replica.cc:442] T a7d0ec2d54174d23854520fe1431dd31 P 0cda85198f024d738628555ffa00ac19: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:24:44.041954 14436 tablet_replica.cc:439] T a7d0ec2d54174d23854520fe1431dd31 P 2af2232dfff944a9aceafbbd871bdd14: TxnStatusTablet state changed. Reason: New leader 696b0b38fed74ae1ba7e427b3c138be5. Latest consensus state: current_term: 1 leader_uuid: "696b0b38fed74ae1ba7e427b3c138be5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } } }
I20260923 17:24:44.042061 14436 tablet_replica.cc:442] T a7d0ec2d54174d23854520fe1431dd31 P 2af2232dfff944a9aceafbbd871bdd14: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:24:44.045302 14451 tablet_replica.cc:439] T a7d0ec2d54174d23854520fe1431dd31 P 696b0b38fed74ae1ba7e427b3c138be5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "696b0b38fed74ae1ba7e427b3c138be5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } } }
I20260923 17:24:44.045439 14451 tablet_replica.cc:442] T a7d0ec2d54174d23854520fe1431dd31 P 696b0b38fed74ae1ba7e427b3c138be5: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:24:44.045589 14451 tablet_replica.cc:439] T a7d0ec2d54174d23854520fe1431dd31 P 696b0b38fed74ae1ba7e427b3c138be5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "696b0b38fed74ae1ba7e427b3c138be5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } } }
I20260923 17:24:44.045666 14451 tablet_replica.cc:442] T a7d0ec2d54174d23854520fe1431dd31 P 696b0b38fed74ae1ba7e427b3c138be5: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:24:44.045856 14434 tablet_replica.cc:439] T a7d0ec2d54174d23854520fe1431dd31 P 0cda85198f024d738628555ffa00ac19: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "696b0b38fed74ae1ba7e427b3c138be5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } } }
I20260923 17:24:44.045960 14434 tablet_replica.cc:442] T a7d0ec2d54174d23854520fe1431dd31 P 0cda85198f024d738628555ffa00ac19: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:24:44.047127 14436 tablet_replica.cc:439] T a7d0ec2d54174d23854520fe1431dd31 P 2af2232dfff944a9aceafbbd871bdd14: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "696b0b38fed74ae1ba7e427b3c138be5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } } }
I20260923 17:24:44.047241 14436 tablet_replica.cc:442] T a7d0ec2d54174d23854520fe1431dd31 P 2af2232dfff944a9aceafbbd871bdd14: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:24:44.048347 14457 mvcc.cc:205] Tried to move back new op lower bound from 7332594827410169856 to 7332594827298824192. Current Snapshot: MvccSnapshot[applied={T|T < 7332594827410169856 or (T in {7332594827410169856})}]
I20260923 17:24:44.048686 14458 mvcc.cc:205] Tried to move back new op lower bound from 7332594827410169856 to 7332594827298824192. Current Snapshot: MvccSnapshot[applied={T|T < 7332594827410169856}]
I20260923 17:24:44.049217 14456 mvcc.cc:205] Tried to move back new op lower bound from 7332594827410169856 to 7332594827298824192. Current Snapshot: MvccSnapshot[applied={T|T < 7332594827410169856}]
W20260923 17:24:44.054286 14126 tablet.cc:2369] T 28e444192eb1486ea5bc8c56e7bc3a0c P 696b0b38fed74ae1ba7e427b3c138be5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:24:44.061887 14425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:44.058046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33554 (local address 127.11.229.2:39963)
0923 17:24:44.058204 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:44.058211 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:44.058228 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:24:44.058440 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:44.058442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:44.058495 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:44.058606 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:24:44.058840 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.059403 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.060355 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.060590 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.060701 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:44.060979 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:24:44.061487 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:24:44.061619 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:44.061689 (+ 70us) server_negotiation.cc:299] Negotiation successful
0923 17:24:44.061742 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 17:24:44.073374 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:44.069787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:33493 (local address 127.11.229.2:43467)
0923 17:24:44.070115 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:44.070132 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:44.070273 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:44.070530 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:44.070533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:44.070554 (+ 21us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:24:44.070660 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:44.070665 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.071413 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.071419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:44.072054 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:44.072062 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.072548 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.072552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:44.072645 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:44.072688 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:24:44.073026 (+ 338us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:24:44.073034 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 17:24:44.073124 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 17:24:44.073207 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":114,"threads_started":1}
I20260923 17:24:44.073365 14427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:44.069955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:43467 (local address 127.11.229.1:33493)
0923 17:24:44.070105 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:44.070110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:44.070214 (+ 104us) server_negotiation.cc:407] Connection header received
0923 17:24:44.070379 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:44.070382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:44.070439 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:44.070519 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:24:44.070767 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.071304 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.072176 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.072414 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.072530 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:44.072849 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:24:44.072942 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:24:44.073023 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:44.073137 (+ 114us) server_negotiation.cc:299] Negotiation successful
0923 17:24:44.073192 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 17:24:44.081518 14359 raft_consensus.cc:1261] T 28e444192eb1486ea5bc8c56e7bc3a0c P 2af2232dfff944a9aceafbbd871bdd14 [term 1 FOLLOWER]: Refusing update from remote peer 0cda85198f024d738628555ffa00ac19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:24:44.081801 14073 raft_consensus.cc:1261] T 28e444192eb1486ea5bc8c56e7bc3a0c P 696b0b38fed74ae1ba7e427b3c138be5 [term 1 FOLLOWER]: Refusing update from remote peer 0cda85198f024d738628555ffa00ac19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:24:44.081969 14434 consensus_queue.cc:1035] T 28e444192eb1486ea5bc8c56e7bc3a0c P 0cda85198f024d738628555ffa00ac19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:24:44.082325 14438 consensus_queue.cc:1035] T 28e444192eb1486ea5bc8c56e7bc3a0c P 0cda85198f024d738628555ffa00ac19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:24:44.106261 14425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:44.102052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:50629 (local address 127.11.229.2:39963)
0923 17:24:44.102175 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:44.102180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:44.102311 (+ 131us) server_negotiation.cc:407] Connection header received
0923 17:24:44.102681 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:44.102686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:44.102746 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:44.102808 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:24:44.103137 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.103658 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.104951 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.105321 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.105397 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:44.105824 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:24:44.105908 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:24:44.105951 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:44.106109 (+ 158us) server_negotiation.cc:299] Negotiation successful
0923 17:24:44.106157 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 17:24:44.106536 14472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:44.101688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:39963 (local address 127.11.229.1:50629)
0923 17:24:44.102216 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:44.102231 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:44.102534 (+ 303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:44.102821 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:44.102824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:44.102843 (+ 19us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:24:44.102973 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:44.102979 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.103771 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.103775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:44.104812 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:44.104824 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.105518 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.105521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:44.105625 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:44.105676 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:24:44.106011 (+ 335us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:24:44.106016 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 17:24:44.106079 (+ 63us) client_negotiation.cc:240] Negotiation successful
0923 17:24:44.106158 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":101,"threads_started":1}
I20260923 17:24:44.135735 13918 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:24:44.135948 13918 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:24:44.138800 13918 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a52102167344208bc65d3c5369023c1: Sending DeleteTablet for 3 replicas of tablet 28e444192eb1486ea5bc8c56e7bc3a0c
I20260923 17:24:44.139557 14339 tablet_service.cc:1514] Processing DeleteTablet for tablet 28e444192eb1486ea5bc8c56e7bc3a0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:24:44 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:55848
I20260923 17:24:44.139814 14199 tablet_service.cc:1514] Processing DeleteTablet for tablet 28e444192eb1486ea5bc8c56e7bc3a0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:24:44 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:33552
I20260923 17:24:44.140333 14476 tablet_replica.cc:330] T 28e444192eb1486ea5bc8c56e7bc3a0c P 2af2232dfff944a9aceafbbd871bdd14: stopping tablet replica
I20260923 17:24:44.140681 14046 tablet_service.cc:1514] Processing DeleteTablet for tablet 28e444192eb1486ea5bc8c56e7bc3a0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:24:44 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:32782
I20260923 17:24:44.140769 14476 raft_consensus.cc:2229] T 28e444192eb1486ea5bc8c56e7bc3a0c P 2af2232dfff944a9aceafbbd871bdd14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:24:44.141106 14476 raft_consensus.cc:2258] T 28e444192eb1486ea5bc8c56e7bc3a0c P 2af2232dfff944a9aceafbbd871bdd14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:24:44.142138 13922 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42312:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:24:44.142590 13922 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:24:44.143180 14476 ts_tablet_manager.cc:1905] T 28e444192eb1486ea5bc8c56e7bc3a0c P 2af2232dfff944a9aceafbbd871bdd14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:24:44.144006 14478 tablet_replica.cc:330] T 28e444192eb1486ea5bc8c56e7bc3a0c P 696b0b38fed74ae1ba7e427b3c138be5: stopping tablet replica
I20260923 17:24:44.144313 14478 raft_consensus.cc:2229] T 28e444192eb1486ea5bc8c56e7bc3a0c P 696b0b38fed74ae1ba7e427b3c138be5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:24:44.144503 14478 raft_consensus.cc:2258] T 28e444192eb1486ea5bc8c56e7bc3a0c P 696b0b38fed74ae1ba7e427b3c138be5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:24:44.144804 14477 tablet_replica.cc:330] T 28e444192eb1486ea5bc8c56e7bc3a0c P 0cda85198f024d738628555ffa00ac19: stopping tablet replica
I20260923 17:24:44.145082 14477 raft_consensus.cc:2229] T 28e444192eb1486ea5bc8c56e7bc3a0c P 0cda85198f024d738628555ffa00ac19 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:24:44.145452 14477 raft_consensus.cc:2258] T 28e444192eb1486ea5bc8c56e7bc3a0c P 0cda85198f024d738628555ffa00ac19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:24:44.146304 14478 ts_tablet_manager.cc:1905] T 28e444192eb1486ea5bc8c56e7bc3a0c P 696b0b38fed74ae1ba7e427b3c138be5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:24:44.148842 14046 tablet_service.cc:1467] Processing CreateTablet for tablet 420ea2ea9d8f42a981d6d9d79383a110 (DEFAULT_TABLE table=test-table [id=f75d0e40ac36469cbd0a9e33ce35f194]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:24:44.149103 14046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 420ea2ea9d8f42a981d6d9d79383a110. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:44.149457 14478 ts_tablet_manager.cc:1918] T 28e444192eb1486ea5bc8c56e7bc3a0c P 696b0b38fed74ae1ba7e427b3c138be5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:24:44.149556 14478 log.cc:1196] T 28e444192eb1486ea5bc8c56e7bc3a0c P 696b0b38fed74ae1ba7e427b3c138be5: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-0/wal/wals/28e444192eb1486ea5bc8c56e7bc3a0c
I20260923 17:24:44.149892 14478 ts_tablet_manager.cc:1939] T 28e444192eb1486ea5bc8c56e7bc3a0c P 696b0b38fed74ae1ba7e427b3c138be5: Deleting consensus metadata
I20260923 17:24:44.149907 14476 ts_tablet_manager.cc:1918] T 28e444192eb1486ea5bc8c56e7bc3a0c P 2af2232dfff944a9aceafbbd871bdd14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:24:44.150056 14476 log.cc:1196] T 28e444192eb1486ea5bc8c56e7bc3a0c P 2af2232dfff944a9aceafbbd871bdd14: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-2/wal/wals/28e444192eb1486ea5bc8c56e7bc3a0c
I20260923 17:24:44.150415 14476 ts_tablet_manager.cc:1939] T 28e444192eb1486ea5bc8c56e7bc3a0c P 2af2232dfff944a9aceafbbd871bdd14: Deleting consensus metadata
I20260923 17:24:44.151538 14199 tablet_service.cc:1467] Processing CreateTablet for tablet 420ea2ea9d8f42a981d6d9d79383a110 (DEFAULT_TABLE table=test-table [id=f75d0e40ac36469cbd0a9e33ce35f194]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:24:44.151813 14199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 420ea2ea9d8f42a981d6d9d79383a110. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:44.151825 14428 tablet_bootstrap.cc:492] T 420ea2ea9d8f42a981d6d9d79383a110 P 696b0b38fed74ae1ba7e427b3c138be5: Bootstrap starting.
I20260923 17:24:44.152791 14428 tablet_bootstrap.cc:654] T 420ea2ea9d8f42a981d6d9d79383a110 P 696b0b38fed74ae1ba7e427b3c138be5: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:44.153633 14339 tablet_service.cc:1467] Processing CreateTablet for tablet 420ea2ea9d8f42a981d6d9d79383a110 (DEFAULT_TABLE table=test-table [id=f75d0e40ac36469cbd0a9e33ce35f194]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:24:44.153882 14339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 420ea2ea9d8f42a981d6d9d79383a110. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:44.154333 14429 tablet_bootstrap.cc:492] T 420ea2ea9d8f42a981d6d9d79383a110 P 0cda85198f024d738628555ffa00ac19: Bootstrap starting.
I20260923 17:24:44.154502 14477 ts_tablet_manager.cc:1905] T 28e444192eb1486ea5bc8c56e7bc3a0c P 0cda85198f024d738628555ffa00ac19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:24:44.155205 14429 tablet_bootstrap.cc:654] T 420ea2ea9d8f42a981d6d9d79383a110 P 0cda85198f024d738628555ffa00ac19: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:44.156059 13907 catalog_manager.cc:4841] TS 2af2232dfff944a9aceafbbd871bdd14 (127.11.229.3:33357): tablet 28e444192eb1486ea5bc8c56e7bc3a0c (table test-table [id=917aa1f81730469daa7dcb9ea261284a]) successfully deleted
I20260923 17:24:44.156396 14430 tablet_bootstrap.cc:492] T 420ea2ea9d8f42a981d6d9d79383a110 P 2af2232dfff944a9aceafbbd871bdd14: Bootstrap starting.
I20260923 17:24:44.156865 14429 tablet_bootstrap.cc:492] T 420ea2ea9d8f42a981d6d9d79383a110 P 0cda85198f024d738628555ffa00ac19: No bootstrap required, opened a new log
I20260923 17:24:44.156934 14429 ts_tablet_manager.cc:1397] T 420ea2ea9d8f42a981d6d9d79383a110 P 0cda85198f024d738628555ffa00ac19: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260923 17:24:44.157377 14430 tablet_bootstrap.cc:654] T 420ea2ea9d8f42a981d6d9d79383a110 P 2af2232dfff944a9aceafbbd871bdd14: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:44.157392 14429 raft_consensus.cc:348] T 420ea2ea9d8f42a981d6d9d79383a110 P 0cda85198f024d738628555ffa00ac19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } }
I20260923 17:24:44.157500 14429 raft_consensus.cc:374] T 420ea2ea9d8f42a981d6d9d79383a110 P 0cda85198f024d738628555ffa00ac19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:44.157532 14429 raft_consensus.cc:729] T 420ea2ea9d8f42a981d6d9d79383a110 P 0cda85198f024d738628555ffa00ac19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cda85198f024d738628555ffa00ac19, State: Initialized, Role: FOLLOWER
I20260923 17:24:44.157677 14429 consensus_queue.cc:260] T 420ea2ea9d8f42a981d6d9d79383a110 P 0cda85198f024d738628555ffa00ac19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } }
I20260923 17:24:44.157940 14429 ts_tablet_manager.cc:1428] T 420ea2ea9d8f42a981d6d9d79383a110 P 0cda85198f024d738628555ffa00ac19: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 17:24:44.158027 14428 tablet_bootstrap.cc:492] T 420ea2ea9d8f42a981d6d9d79383a110 P 696b0b38fed74ae1ba7e427b3c138be5: No bootstrap required, opened a new log
I20260923 17:24:44.158116 14428 ts_tablet_manager.cc:1397] T 420ea2ea9d8f42a981d6d9d79383a110 P 696b0b38fed74ae1ba7e427b3c138be5: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260923 17:24:44.158581 14428 raft_consensus.cc:348] T 420ea2ea9d8f42a981d6d9d79383a110 P 696b0b38fed74ae1ba7e427b3c138be5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } }
I20260923 17:24:44.158635 14477 ts_tablet_manager.cc:1918] T 28e444192eb1486ea5bc8c56e7bc3a0c P 0cda85198f024d738628555ffa00ac19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:24:44.158689 14428 raft_consensus.cc:374] T 420ea2ea9d8f42a981d6d9d79383a110 P 696b0b38fed74ae1ba7e427b3c138be5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:44.158716 14428 raft_consensus.cc:729] T 420ea2ea9d8f42a981d6d9d79383a110 P 696b0b38fed74ae1ba7e427b3c138be5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696b0b38fed74ae1ba7e427b3c138be5, State: Initialized, Role: FOLLOWER
I20260923 17:24:44.158725 14477 log.cc:1196] T 28e444192eb1486ea5bc8c56e7bc3a0c P 0cda85198f024d738628555ffa00ac19: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-1/wal/wals/28e444192eb1486ea5bc8c56e7bc3a0c
I20260923 17:24:44.158828 14428 consensus_queue.cc:260] T 420ea2ea9d8f42a981d6d9d79383a110 P 696b0b38fed74ae1ba7e427b3c138be5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } }
I20260923 17:24:44.159060 14477 ts_tablet_manager.cc:1939] T 28e444192eb1486ea5bc8c56e7bc3a0c P 0cda85198f024d738628555ffa00ac19: Deleting consensus metadata
I20260923 17:24:44.159051 13907 catalog_manager.cc:4841] TS 696b0b38fed74ae1ba7e427b3c138be5 (127.11.229.1:33493): tablet 28e444192eb1486ea5bc8c56e7bc3a0c (table test-table [id=917aa1f81730469daa7dcb9ea261284a]) successfully deleted
I20260923 17:24:44.159215 14428 ts_tablet_manager.cc:1428] T 420ea2ea9d8f42a981d6d9d79383a110 P 696b0b38fed74ae1ba7e427b3c138be5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 17:24:44.160514 13907 catalog_manager.cc:4841] TS 0cda85198f024d738628555ffa00ac19 (127.11.229.2:39963): tablet 28e444192eb1486ea5bc8c56e7bc3a0c (table test-table [id=917aa1f81730469daa7dcb9ea261284a]) successfully deleted
I20260923 17:24:44.162726 14430 tablet_bootstrap.cc:492] T 420ea2ea9d8f42a981d6d9d79383a110 P 2af2232dfff944a9aceafbbd871bdd14: No bootstrap required, opened a new log
I20260923 17:24:44.162820 14430 ts_tablet_manager.cc:1397] T 420ea2ea9d8f42a981d6d9d79383a110 P 2af2232dfff944a9aceafbbd871bdd14: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260923 17:24:44.163566 14430 raft_consensus.cc:348] T 420ea2ea9d8f42a981d6d9d79383a110 P 2af2232dfff944a9aceafbbd871bdd14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } }
I20260923 17:24:44.163687 14430 raft_consensus.cc:374] T 420ea2ea9d8f42a981d6d9d79383a110 P 2af2232dfff944a9aceafbbd871bdd14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:44.163723 14430 raft_consensus.cc:729] T 420ea2ea9d8f42a981d6d9d79383a110 P 2af2232dfff944a9aceafbbd871bdd14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2af2232dfff944a9aceafbbd871bdd14, State: Initialized, Role: FOLLOWER
I20260923 17:24:44.163870 14430 consensus_queue.cc:260] T 420ea2ea9d8f42a981d6d9d79383a110 P 2af2232dfff944a9aceafbbd871bdd14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } }
I20260923 17:24:44.164640 14430 ts_tablet_manager.cc:1428] T 420ea2ea9d8f42a981d6d9d79383a110 P 2af2232dfff944a9aceafbbd871bdd14: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260923 17:24:44.333447 14435 raft_consensus.cc:482] T 420ea2ea9d8f42a981d6d9d79383a110 P 696b0b38fed74ae1ba7e427b3c138be5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:24:44.333603 14435 raft_consensus.cc:504] T 420ea2ea9d8f42a981d6d9d79383a110 P 696b0b38fed74ae1ba7e427b3c138be5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } }
I20260923 17:24:44.334085 14435 leader_election.cc:288] T 420ea2ea9d8f42a981d6d9d79383a110 P 696b0b38fed74ae1ba7e427b3c138be5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cda85198f024d738628555ffa00ac19 (127.11.229.2:39963), 2af2232dfff944a9aceafbbd871bdd14 (127.11.229.3:33357)
I20260923 17:24:44.334640 14359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420ea2ea9d8f42a981d6d9d79383a110" candidate_uuid: "696b0b38fed74ae1ba7e427b3c138be5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2af2232dfff944a9aceafbbd871bdd14" is_pre_election: true
I20260923 17:24:44.334640 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420ea2ea9d8f42a981d6d9d79383a110" candidate_uuid: "696b0b38fed74ae1ba7e427b3c138be5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cda85198f024d738628555ffa00ac19" is_pre_election: true
I20260923 17:24:44.334798 14359 raft_consensus.cc:2454] T 420ea2ea9d8f42a981d6d9d79383a110 P 2af2232dfff944a9aceafbbd871bdd14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696b0b38fed74ae1ba7e427b3c138be5 in term 0.
I20260923 17:24:44.334798 14219 raft_consensus.cc:2454] T 420ea2ea9d8f42a981d6d9d79383a110 P 0cda85198f024d738628555ffa00ac19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696b0b38fed74ae1ba7e427b3c138be5 in term 0.
I20260923 17:24:44.335126 14011 leader_election.cc:302] T 420ea2ea9d8f42a981d6d9d79383a110 P 696b0b38fed74ae1ba7e427b3c138be5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2af2232dfff944a9aceafbbd871bdd14, 696b0b38fed74ae1ba7e427b3c138be5; no voters:
I20260923 17:24:44.335307 14435 raft_consensus.cc:2781] T 420ea2ea9d8f42a981d6d9d79383a110 P 696b0b38fed74ae1ba7e427b3c138be5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:24:44.335367 14435 raft_consensus.cc:482] T 420ea2ea9d8f42a981d6d9d79383a110 P 696b0b38fed74ae1ba7e427b3c138be5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:24:44.335408 14435 raft_consensus.cc:3037] T 420ea2ea9d8f42a981d6d9d79383a110 P 696b0b38fed74ae1ba7e427b3c138be5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:44.336400 14435 raft_consensus.cc:504] T 420ea2ea9d8f42a981d6d9d79383a110 P 696b0b38fed74ae1ba7e427b3c138be5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } }
I20260923 17:24:44.336726 14435 leader_election.cc:288] T 420ea2ea9d8f42a981d6d9d79383a110 P 696b0b38fed74ae1ba7e427b3c138be5 [CANDIDATE]: Term 1 election: Requested vote from peers 0cda85198f024d738628555ffa00ac19 (127.11.229.2:39963), 2af2232dfff944a9aceafbbd871bdd14 (127.11.229.3:33357)
I20260923 17:24:44.337169 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420ea2ea9d8f42a981d6d9d79383a110" candidate_uuid: "696b0b38fed74ae1ba7e427b3c138be5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cda85198f024d738628555ffa00ac19"
I20260923 17:24:44.337175 14359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420ea2ea9d8f42a981d6d9d79383a110" candidate_uuid: "696b0b38fed74ae1ba7e427b3c138be5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2af2232dfff944a9aceafbbd871bdd14"
I20260923 17:24:44.337332 14219 raft_consensus.cc:3037] T 420ea2ea9d8f42a981d6d9d79383a110 P 0cda85198f024d738628555ffa00ac19 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:44.337332 14359 raft_consensus.cc:3037] T 420ea2ea9d8f42a981d6d9d79383a110 P 2af2232dfff944a9aceafbbd871bdd14 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:44.338501 14219 raft_consensus.cc:2454] T 420ea2ea9d8f42a981d6d9d79383a110 P 0cda85198f024d738628555ffa00ac19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696b0b38fed74ae1ba7e427b3c138be5 in term 1.
I20260923 17:24:44.338501 14359 raft_consensus.cc:2454] T 420ea2ea9d8f42a981d6d9d79383a110 P 2af2232dfff944a9aceafbbd871bdd14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696b0b38fed74ae1ba7e427b3c138be5 in term 1.
I20260923 17:24:44.338814 14011 leader_election.cc:302] T 420ea2ea9d8f42a981d6d9d79383a110 P 696b0b38fed74ae1ba7e427b3c138be5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2af2232dfff944a9aceafbbd871bdd14, 696b0b38fed74ae1ba7e427b3c138be5; no voters:
I20260923 17:24:44.339001 14435 raft_consensus.cc:2781] T 420ea2ea9d8f42a981d6d9d79383a110 P 696b0b38fed74ae1ba7e427b3c138be5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:24:44.339098 14435 raft_consensus.cc:686] T 420ea2ea9d8f42a981d6d9d79383a110 P 696b0b38fed74ae1ba7e427b3c138be5 [term 1 LEADER]: Becoming Leader. State: Replica: 696b0b38fed74ae1ba7e427b3c138be5, State: Running, Role: LEADER
I20260923 17:24:44.339247 14435 consensus_queue.cc:237] T 420ea2ea9d8f42a981d6d9d79383a110 P 696b0b38fed74ae1ba7e427b3c138be5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } }
I20260923 17:24:44.341017 13921 catalog_manager.cc:5495] T 420ea2ea9d8f42a981d6d9d79383a110 P 696b0b38fed74ae1ba7e427b3c138be5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696b0b38fed74ae1ba7e427b3c138be5 (127.11.229.1). New cstate: current_term: 1 leader_uuid: "696b0b38fed74ae1ba7e427b3c138be5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } health_report { overall_health: UNKNOWN } } }
I20260923 17:24:44.363375 14427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:44.360338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50590 (local address 127.11.229.1:33493)
0923 17:24:44.360469 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:44.360473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:44.360486 (+ 13us) server_negotiation.cc:407] Connection header received
0923 17:24:44.360592 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:44.360594 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:44.360640 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:44.360706 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:24:44.360917 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.361521 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.362339 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.362571 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.362651 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:44.362938 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:24:44.363011 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:24:44.363086 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:44.363216 (+ 130us) server_negotiation.cc:299] Negotiation successful
0923 17:24:44.363264 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 17:24:44.372040 14427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:44.369351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:49371 (local address 127.11.229.1:33493)
0923 17:24:44.369460 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:44.369463 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:44.369475 (+ 12us) server_negotiation.cc:407] Connection header received
0923 17:24:44.369579 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:44.369581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:44.369619 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:44.369682 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:24:44.369904 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.370376 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.371197 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.371394 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.371467 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:44.371670 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:24:44.371731 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:24:44.371783 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:44.371864 (+ 81us) server_negotiation.cc:299] Negotiation successful
0923 17:24:44.371906 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":40}
I20260923 17:24:44.372054 14472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:44.369237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:33493 (local address 127.11.229.1:49371)
0923 17:24:44.369416 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:44.369430 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:44.369492 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:44.369711 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:44.369713 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:44.369718 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:24:44.369816 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:44.369821 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.370488 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.370491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:44.371097 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:44.371105 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:44.371488 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:44.371490 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:44.371550 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:44.371574 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:24:44.371804 (+ 230us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:24:44.371809 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 17:24:44.371857 (+ 48us) client_negotiation.cc:240] Negotiation successful
0923 17:24:44.371923 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":14}
I20260923 17:24:44.377677 14359 raft_consensus.cc:1261] T 420ea2ea9d8f42a981d6d9d79383a110 P 2af2232dfff944a9aceafbbd871bdd14 [term 1 FOLLOWER]: Refusing update from remote peer 696b0b38fed74ae1ba7e427b3c138be5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:24:44.377732 14219 raft_consensus.cc:1261] T 420ea2ea9d8f42a981d6d9d79383a110 P 0cda85198f024d738628555ffa00ac19 [term 1 FOLLOWER]: Refusing update from remote peer 696b0b38fed74ae1ba7e427b3c138be5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:24:44.378120 14451 consensus_queue.cc:1035] T 420ea2ea9d8f42a981d6d9d79383a110 P 696b0b38fed74ae1ba7e427b3c138be5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:24:44.378275 14435 consensus_queue.cc:1035] T 420ea2ea9d8f42a981d6d9d79383a110 P 696b0b38fed74ae1ba7e427b3c138be5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 17:24:44.392540 14456 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 17:24:45.948298 13922 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:24:45.948500 13922 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:24:45.951105 13922 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a52102167344208bc65d3c5369023c1: Sending DeleteTablet for 3 replicas of tablet 420ea2ea9d8f42a981d6d9d79383a110
I20260923 17:24:45.951690 14046 tablet_service.cc:1514] Processing DeleteTablet for tablet 420ea2ea9d8f42a981d6d9d79383a110 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:24:45 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:32782
I20260923 17:24:45.951828 14199 tablet_service.cc:1514] Processing DeleteTablet for tablet 420ea2ea9d8f42a981d6d9d79383a110 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:24:45 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:33552
I20260923 17:24:45.952092 14506 tablet_replica.cc:330] T 420ea2ea9d8f42a981d6d9d79383a110 P 696b0b38fed74ae1ba7e427b3c138be5: stopping tablet replica
I20260923 17:24:45.952237 14506 raft_consensus.cc:2229] T 420ea2ea9d8f42a981d6d9d79383a110 P 696b0b38fed74ae1ba7e427b3c138be5 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:24:45.952239 14339 tablet_service.cc:1514] Processing DeleteTablet for tablet 420ea2ea9d8f42a981d6d9d79383a110 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:24:45 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:55848
I20260923 17:24:45.952565 14508 tablet_replica.cc:330] T 420ea2ea9d8f42a981d6d9d79383a110 P 2af2232dfff944a9aceafbbd871bdd14: stopping tablet replica
I20260923 17:24:45.952597 14506 raft_consensus.cc:2258] T 420ea2ea9d8f42a981d6d9d79383a110 P 696b0b38fed74ae1ba7e427b3c138be5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:24:45.952713 14508 raft_consensus.cc:2229] T 420ea2ea9d8f42a981d6d9d79383a110 P 2af2232dfff944a9aceafbbd871bdd14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:24:45.952823 14508 raft_consensus.cc:2258] T 420ea2ea9d8f42a981d6d9d79383a110 P 2af2232dfff944a9aceafbbd871bdd14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:24:45.953464 14506 ts_tablet_manager.cc:1905] T 420ea2ea9d8f42a981d6d9d79383a110 P 696b0b38fed74ae1ba7e427b3c138be5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:24:45.953516 14508 ts_tablet_manager.cc:1905] T 420ea2ea9d8f42a981d6d9d79383a110 P 2af2232dfff944a9aceafbbd871bdd14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:24:45.955880 14507 tablet_replica.cc:330] T 420ea2ea9d8f42a981d6d9d79383a110 P 0cda85198f024d738628555ffa00ac19: stopping tablet replica
I20260923 17:24:45.956065 14507 raft_consensus.cc:2229] T 420ea2ea9d8f42a981d6d9d79383a110 P 0cda85198f024d738628555ffa00ac19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:24:45.956264 14507 raft_consensus.cc:2258] T 420ea2ea9d8f42a981d6d9d79383a110 P 0cda85198f024d738628555ffa00ac19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:24:45.956820 14508 ts_tablet_manager.cc:1918] T 420ea2ea9d8f42a981d6d9d79383a110 P 2af2232dfff944a9aceafbbd871bdd14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 17:24:45.956902 14508 log.cc:1196] T 420ea2ea9d8f42a981d6d9d79383a110 P 2af2232dfff944a9aceafbbd871bdd14: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-2/wal/wals/420ea2ea9d8f42a981d6d9d79383a110
I20260923 17:24:45.957187 14508 ts_tablet_manager.cc:1939] T 420ea2ea9d8f42a981d6d9d79383a110 P 2af2232dfff944a9aceafbbd871bdd14: Deleting consensus metadata
I20260923 17:24:45.957304 14506 ts_tablet_manager.cc:1918] T 420ea2ea9d8f42a981d6d9d79383a110 P 696b0b38fed74ae1ba7e427b3c138be5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 17:24:45.957324 14507 ts_tablet_manager.cc:1905] T 420ea2ea9d8f42a981d6d9d79383a110 P 0cda85198f024d738628555ffa00ac19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:24:45.957408 14506 log.cc:1196] T 420ea2ea9d8f42a981d6d9d79383a110 P 696b0b38fed74ae1ba7e427b3c138be5: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-0/wal/wals/420ea2ea9d8f42a981d6d9d79383a110
I20260923 17:24:45.957767 14506 ts_tablet_manager.cc:1939] T 420ea2ea9d8f42a981d6d9d79383a110 P 696b0b38fed74ae1ba7e427b3c138be5: Deleting consensus metadata
I20260923 17:24:45.957986 13907 catalog_manager.cc:4841] TS 2af2232dfff944a9aceafbbd871bdd14 (127.11.229.3:33357): tablet 420ea2ea9d8f42a981d6d9d79383a110 (table test-table [id=f75d0e40ac36469cbd0a9e33ce35f194]) successfully deleted
I20260923 17:24:45.958777 13907 catalog_manager.cc:4841] TS 696b0b38fed74ae1ba7e427b3c138be5 (127.11.229.1:33493): tablet 420ea2ea9d8f42a981d6d9d79383a110 (table test-table [id=f75d0e40ac36469cbd0a9e33ce35f194]) successfully deleted
I20260923 17:24:45.960124 14507 ts_tablet_manager.cc:1918] T 420ea2ea9d8f42a981d6d9d79383a110 P 0cda85198f024d738628555ffa00ac19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 17:24:45.960227 14507 log.cc:1196] T 420ea2ea9d8f42a981d6d9d79383a110 P 0cda85198f024d738628555ffa00ac19: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-1/wal/wals/420ea2ea9d8f42a981d6d9d79383a110
I20260923 17:24:45.960508 14507 ts_tablet_manager.cc:1939] T 420ea2ea9d8f42a981d6d9d79383a110 P 0cda85198f024d738628555ffa00ac19: Deleting consensus metadata
I20260923 17:24:45.961292 13907 catalog_manager.cc:4841] TS 0cda85198f024d738628555ffa00ac19 (127.11.229.2:39963): tablet 420ea2ea9d8f42a981d6d9d79383a110 (table test-table [id=f75d0e40ac36469cbd0a9e33ce35f194]) successfully deleted
Sep 23 17:24:45 dist-test-slave-ptl9 krb5kdc[13877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184285, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 17:24:45 dist-test-slave-ptl9 krb5kdc[13877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184285, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:24:45.981580 14516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:45.972500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53638 (local address 127.11.229.62:46305)
0923 17:24:45.972791 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:45.972797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:45.972813 (+ 16us) server_negotiation.cc:407] Connection header received
0923 17:24:45.972975 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:45.972978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:45.973025 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:45.973112 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:45.973873 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:45.974539 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:45.975368 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:45.975621 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:45.975706 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:45.978252 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:45.978256 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:45.978258 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:45.978284 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:45.979698 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:45.980156 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:45.980159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:45.980161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:45.980252 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:45.980533 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:45.980536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:45.980538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:45.980787 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:45.980944 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:45.981240 (+ 296us) server_negotiation.cc:299] Negotiation successful
0923 17:24:45.981438 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":126,"threads_started":1}
I20260923 17:24:45.984280 13922 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:24:45.984730 13922 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:24:45.989696 14046 tablet_service.cc:1467] Processing CreateTablet for tablet 62f5086166d14d2d977885cecb9d359d (DEFAULT_TABLE table=test-table [id=1f1b5cf787cf4aabb779c5551794ffb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:24:45.989696 14199 tablet_service.cc:1467] Processing CreateTablet for tablet 62f5086166d14d2d977885cecb9d359d (DEFAULT_TABLE table=test-table [id=1f1b5cf787cf4aabb779c5551794ffb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:24:45.989882 14339 tablet_service.cc:1467] Processing CreateTablet for tablet 62f5086166d14d2d977885cecb9d359d (DEFAULT_TABLE table=test-table [id=1f1b5cf787cf4aabb779c5551794ffb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:24:45.990026 14046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62f5086166d14d2d977885cecb9d359d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:45.990073 14199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62f5086166d14d2d977885cecb9d359d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:45.990135 14339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62f5086166d14d2d977885cecb9d359d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:45.992388 14517 tablet_bootstrap.cc:492] T 62f5086166d14d2d977885cecb9d359d P 0cda85198f024d738628555ffa00ac19: Bootstrap starting.
I20260923 17:24:45.992923 14519 tablet_bootstrap.cc:492] T 62f5086166d14d2d977885cecb9d359d P 2af2232dfff944a9aceafbbd871bdd14: Bootstrap starting.
I20260923 17:24:45.993427 14517 tablet_bootstrap.cc:654] T 62f5086166d14d2d977885cecb9d359d P 0cda85198f024d738628555ffa00ac19: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:45.993489 14518 tablet_bootstrap.cc:492] T 62f5086166d14d2d977885cecb9d359d P 696b0b38fed74ae1ba7e427b3c138be5: Bootstrap starting.
I20260923 17:24:45.993856 14519 tablet_bootstrap.cc:654] T 62f5086166d14d2d977885cecb9d359d P 2af2232dfff944a9aceafbbd871bdd14: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:45.994457 14518 tablet_bootstrap.cc:654] T 62f5086166d14d2d977885cecb9d359d P 696b0b38fed74ae1ba7e427b3c138be5: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:45.994784 14517 tablet_bootstrap.cc:492] T 62f5086166d14d2d977885cecb9d359d P 0cda85198f024d738628555ffa00ac19: No bootstrap required, opened a new log
I20260923 17:24:45.994881 14517 ts_tablet_manager.cc:1397] T 62f5086166d14d2d977885cecb9d359d P 0cda85198f024d738628555ffa00ac19: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 17:24:45.995169 14519 tablet_bootstrap.cc:492] T 62f5086166d14d2d977885cecb9d359d P 2af2232dfff944a9aceafbbd871bdd14: No bootstrap required, opened a new log
I20260923 17:24:45.995251 14519 ts_tablet_manager.cc:1397] T 62f5086166d14d2d977885cecb9d359d P 2af2232dfff944a9aceafbbd871bdd14: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 17:24:45.995409 14517 raft_consensus.cc:348] T 62f5086166d14d2d977885cecb9d359d P 0cda85198f024d738628555ffa00ac19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } }
I20260923 17:24:45.995533 14517 raft_consensus.cc:374] T 62f5086166d14d2d977885cecb9d359d P 0cda85198f024d738628555ffa00ac19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:45.995568 14517 raft_consensus.cc:729] T 62f5086166d14d2d977885cecb9d359d P 0cda85198f024d738628555ffa00ac19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cda85198f024d738628555ffa00ac19, State: Initialized, Role: FOLLOWER
I20260923 17:24:45.995698 14517 consensus_queue.cc:260] T 62f5086166d14d2d977885cecb9d359d P 0cda85198f024d738628555ffa00ac19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } }
I20260923 17:24:45.995716 14519 raft_consensus.cc:348] T 62f5086166d14d2d977885cecb9d359d P 2af2232dfff944a9aceafbbd871bdd14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } }
I20260923 17:24:45.995810 14519 raft_consensus.cc:374] T 62f5086166d14d2d977885cecb9d359d P 2af2232dfff944a9aceafbbd871bdd14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:45.995842 14519 raft_consensus.cc:729] T 62f5086166d14d2d977885cecb9d359d P 2af2232dfff944a9aceafbbd871bdd14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2af2232dfff944a9aceafbbd871bdd14, State: Initialized, Role: FOLLOWER
I20260923 17:24:45.995990 14519 consensus_queue.cc:260] T 62f5086166d14d2d977885cecb9d359d P 2af2232dfff944a9aceafbbd871bdd14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } }
I20260923 17:24:45.996112 14517 ts_tablet_manager.cc:1428] T 62f5086166d14d2d977885cecb9d359d P 0cda85198f024d738628555ffa00ac19: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 17:24:45.996357 14519 ts_tablet_manager.cc:1428] T 62f5086166d14d2d977885cecb9d359d P 2af2232dfff944a9aceafbbd871bdd14: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 17:24:45.997382 14518 tablet_bootstrap.cc:492] T 62f5086166d14d2d977885cecb9d359d P 696b0b38fed74ae1ba7e427b3c138be5: No bootstrap required, opened a new log
I20260923 17:24:45.997502 14518 ts_tablet_manager.cc:1397] T 62f5086166d14d2d977885cecb9d359d P 696b0b38fed74ae1ba7e427b3c138be5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260923 17:24:45.997987 14518 raft_consensus.cc:348] T 62f5086166d14d2d977885cecb9d359d P 696b0b38fed74ae1ba7e427b3c138be5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } }
I20260923 17:24:45.998117 14518 raft_consensus.cc:374] T 62f5086166d14d2d977885cecb9d359d P 696b0b38fed74ae1ba7e427b3c138be5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:45.998155 14518 raft_consensus.cc:729] T 62f5086166d14d2d977885cecb9d359d P 696b0b38fed74ae1ba7e427b3c138be5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696b0b38fed74ae1ba7e427b3c138be5, State: Initialized, Role: FOLLOWER
I20260923 17:24:45.998284 14518 consensus_queue.cc:260] T 62f5086166d14d2d977885cecb9d359d P 696b0b38fed74ae1ba7e427b3c138be5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } }
I20260923 17:24:45.998646 14518 ts_tablet_manager.cc:1428] T 62f5086166d14d2d977885cecb9d359d P 696b0b38fed74ae1ba7e427b3c138be5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 17:24:46.314353 14524 raft_consensus.cc:482] T 62f5086166d14d2d977885cecb9d359d P 2af2232dfff944a9aceafbbd871bdd14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:24:46.314510 14524 raft_consensus.cc:504] T 62f5086166d14d2d977885cecb9d359d P 2af2232dfff944a9aceafbbd871bdd14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } }
I20260923 17:24:46.316004 14524 leader_election.cc:288] T 62f5086166d14d2d977885cecb9d359d P 2af2232dfff944a9aceafbbd871bdd14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 696b0b38fed74ae1ba7e427b3c138be5 (127.11.229.1:33493), 0cda85198f024d738628555ffa00ac19 (127.11.229.2:39963)
I20260923 17:24:46.322067 14528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:46.317023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:56303 (local address 127.11.229.1:33493)
0923 17:24:46.317332 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:46.317341 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:46.317356 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:24:46.317410 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:46.317413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:46.317460 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:46.317555 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:24:46.318005 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:46.318823 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:46.320643 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:46.321309 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:46.321444 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:46.321507 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:46.321840 (+ 333us) server_negotiation.cc:299] Negotiation successful
0923 17:24:46.321918 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":113,"threads_started":1}
I20260923 17:24:46.322067 14527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:46.316792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:54043 (local address 127.11.229.2:39963)
0923 17:24:46.317110 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:46.317127 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:46.317275 (+ 148us) server_negotiation.cc:407] Connection header received
0923 17:24:46.317412 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:46.317415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:46.317460 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:46.317554 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:24:46.318038 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:46.318823 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:46.320644 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:46.321341 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:46.321445 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:46.321500 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:46.321840 (+ 340us) server_negotiation.cc:299] Negotiation successful
0923 17:24:46.321918 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":112,"threads_started":1}
I20260923 17:24:46.322135 14525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:46.316222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:33493 (local address 127.11.229.3:56303)
0923 17:24:46.316573 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:46.316592 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:46.316721 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:46.317583 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:46.317586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:46.317630 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:24:46.317877 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:46.317885 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:46.318961 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:46.318964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:46.320503 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:46.320516 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:46.321467 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:46.321470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:46.321662 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:46.321725 (+ 63us) client_negotiation.cc:769] Sending connection context
0923 17:24:46.321809 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 17:24:46.321949 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":293,"spinlock_wait_cycles":5120,"thread_start_us":109,"threads_started":1}
I20260923 17:24:46.322144 14526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:46.316674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:39963 (local address 127.11.229.3:54043)
0923 17:24:46.317156 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:46.317171 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:46.317314 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:46.317583 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:46.317586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:46.317629 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:24:46.317880 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:46.317885 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:46.318945 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:46.318951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:46.320503 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:46.320516 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:46.321456 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:46.321462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:46.321665 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:46.321725 (+ 60us) client_negotiation.cc:769] Sending connection context
0923 17:24:46.321809 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 17:24:46.321949 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":233,"spinlock_wait_cycles":39680,"thread_start_us":96,"threads_started":1}
I20260923 17:24:46.322839 14073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62f5086166d14d2d977885cecb9d359d" candidate_uuid: "2af2232dfff944a9aceafbbd871bdd14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696b0b38fed74ae1ba7e427b3c138be5" is_pre_election: true
I20260923 17:24:46.323032 14073 raft_consensus.cc:2454] T 62f5086166d14d2d977885cecb9d359d P 696b0b38fed74ae1ba7e427b3c138be5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2af2232dfff944a9aceafbbd871bdd14 in term 0.
I20260923 17:24:46.323145 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62f5086166d14d2d977885cecb9d359d" candidate_uuid: "2af2232dfff944a9aceafbbd871bdd14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cda85198f024d738628555ffa00ac19" is_pre_election: true
I20260923 17:24:46.323302 14219 raft_consensus.cc:2454] T 62f5086166d14d2d977885cecb9d359d P 0cda85198f024d738628555ffa00ac19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2af2232dfff944a9aceafbbd871bdd14 in term 0.
I20260923 17:24:46.323490 14291 leader_election.cc:302] T 62f5086166d14d2d977885cecb9d359d P 2af2232dfff944a9aceafbbd871bdd14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2af2232dfff944a9aceafbbd871bdd14, 696b0b38fed74ae1ba7e427b3c138be5; no voters:
I20260923 17:24:46.323755 14524 raft_consensus.cc:2781] T 62f5086166d14d2d977885cecb9d359d P 2af2232dfff944a9aceafbbd871bdd14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:24:46.323814 14524 raft_consensus.cc:482] T 62f5086166d14d2d977885cecb9d359d P 2af2232dfff944a9aceafbbd871bdd14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:24:46.323853 14524 raft_consensus.cc:3037] T 62f5086166d14d2d977885cecb9d359d P 2af2232dfff944a9aceafbbd871bdd14 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:46.325065 14524 raft_consensus.cc:504] T 62f5086166d14d2d977885cecb9d359d P 2af2232dfff944a9aceafbbd871bdd14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } }
I20260923 17:24:46.325498 14524 leader_election.cc:288] T 62f5086166d14d2d977885cecb9d359d P 2af2232dfff944a9aceafbbd871bdd14 [CANDIDATE]: Term 1 election: Requested vote from peers 696b0b38fed74ae1ba7e427b3c138be5 (127.11.229.1:33493), 0cda85198f024d738628555ffa00ac19 (127.11.229.2:39963)
I20260923 17:24:46.325947 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62f5086166d14d2d977885cecb9d359d" candidate_uuid: "2af2232dfff944a9aceafbbd871bdd14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cda85198f024d738628555ffa00ac19"
I20260923 17:24:46.325961 14073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62f5086166d14d2d977885cecb9d359d" candidate_uuid: "2af2232dfff944a9aceafbbd871bdd14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696b0b38fed74ae1ba7e427b3c138be5"
I20260923 17:24:46.326050 14219 raft_consensus.cc:3037] T 62f5086166d14d2d977885cecb9d359d P 0cda85198f024d738628555ffa00ac19 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:46.326143 14073 raft_consensus.cc:3037] T 62f5086166d14d2d977885cecb9d359d P 696b0b38fed74ae1ba7e427b3c138be5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:46.327003 14219 raft_consensus.cc:2454] T 62f5086166d14d2d977885cecb9d359d P 0cda85198f024d738628555ffa00ac19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2af2232dfff944a9aceafbbd871bdd14 in term 1.
I20260923 17:24:46.327152 14073 raft_consensus.cc:2454] T 62f5086166d14d2d977885cecb9d359d P 696b0b38fed74ae1ba7e427b3c138be5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2af2232dfff944a9aceafbbd871bdd14 in term 1.
I20260923 17:24:46.327319 14291 leader_election.cc:302] T 62f5086166d14d2d977885cecb9d359d P 2af2232dfff944a9aceafbbd871bdd14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cda85198f024d738628555ffa00ac19, 2af2232dfff944a9aceafbbd871bdd14; no voters:
I20260923 17:24:46.327497 14524 raft_consensus.cc:2781] T 62f5086166d14d2d977885cecb9d359d P 2af2232dfff944a9aceafbbd871bdd14 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:24:46.327723 14524 raft_consensus.cc:686] T 62f5086166d14d2d977885cecb9d359d P 2af2232dfff944a9aceafbbd871bdd14 [term 1 LEADER]: Becoming Leader. State: Replica: 2af2232dfff944a9aceafbbd871bdd14, State: Running, Role: LEADER
I20260923 17:24:46.328023 14524 consensus_queue.cc:237] T 62f5086166d14d2d977885cecb9d359d P 2af2232dfff944a9aceafbbd871bdd14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } }
I20260923 17:24:46.330825 13921 catalog_manager.cc:5495] T 62f5086166d14d2d977885cecb9d359d P 2af2232dfff944a9aceafbbd871bdd14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2af2232dfff944a9aceafbbd871bdd14 (127.11.229.3). New cstate: current_term: 1 leader_uuid: "2af2232dfff944a9aceafbbd871bdd14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } health_report { overall_health: HEALTHY } } }
I20260923 17:24:46.369098 14532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:46.365128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41006 (local address 127.11.229.3:33357)
0923 17:24:46.365417 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:46.365429 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:46.365444 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:24:46.365536 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:46.365538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:46.365590 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:46.365725 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:24:46.366002 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:46.366546 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:46.367444 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:46.367698 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:46.367817 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:46.368113 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:24:46.368661 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:24:46.368793 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:46.368855 (+ 62us) server_negotiation.cc:299] Negotiation successful
0923 17:24:46.368907 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":108,"threads_started":1}
I20260923 17:24:46.379886 14528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:46.376500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:57779 (local address 127.11.229.1:33493)
0923 17:24:46.376635 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:46.376638 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:46.376727 (+ 89us) server_negotiation.cc:407] Connection header received
0923 17:24:46.376840 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:46.376842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:46.376888 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:46.376952 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:24:46.377249 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:46.377900 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:46.378866 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:46.379087 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:46.379169 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:46.379450 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:24:46.379523 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:24:46.379576 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:46.379656 (+ 80us) server_negotiation.cc:299] Negotiation successful
0923 17:24:46.379701 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 17:24:46.379935 14535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:46.376254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:33493 (local address 127.11.229.3:57779)
0923 17:24:46.376650 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:46.376663 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:46.376751 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:46.376974 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:46.376976 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:46.376983 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:24:46.377135 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:46.377140 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:46.378014 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:46.378017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:46.378754 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:46.378763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:46.379181 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:46.379184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:46.379287 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:46.379332 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:24:46.379595 (+ 263us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:24:46.379600 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 17:24:46.379648 (+ 48us) client_negotiation.cc:240] Negotiation successful
0923 17:24:46.379727 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":119,"threads_started":1}
I20260923 17:24:46.385859 14073 raft_consensus.cc:1261] T 62f5086166d14d2d977885cecb9d359d P 696b0b38fed74ae1ba7e427b3c138be5 [term 1 FOLLOWER]: Refusing update from remote peer 2af2232dfff944a9aceafbbd871bdd14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:24:46.385900 14219 raft_consensus.cc:1261] T 62f5086166d14d2d977885cecb9d359d P 0cda85198f024d738628555ffa00ac19 [term 1 FOLLOWER]: Refusing update from remote peer 2af2232dfff944a9aceafbbd871bdd14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:24:46.386387 14529 consensus_queue.cc:1035] T 62f5086166d14d2d977885cecb9d359d P 2af2232dfff944a9aceafbbd871bdd14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:24:46.386603 14524 consensus_queue.cc:1035] T 62f5086166d14d2d977885cecb9d359d P 2af2232dfff944a9aceafbbd871bdd14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:24:46.405432 14532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:46.402356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:59589 (local address 127.11.229.3:33357)
0923 17:24:46.402477 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:46.402482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:46.402502 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:24:46.402541 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:46.402543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:46.402582 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:46.402644 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:24:46.402862 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:46.403353 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:46.404261 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:46.404510 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:46.404621 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:46.404950 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:24:46.405041 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:24:46.405121 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:46.405218 (+ 97us) server_negotiation.cc:299] Negotiation successful
0923 17:24:46.405307 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 17:24:46.405457 14540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:46.401223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:33357 (local address 127.11.229.1:59589)
0923 17:24:46.401562 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:46.401580 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:46.401680 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:46.402684 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:46.402686 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:46.402694 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:24:46.402791 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:46.402795 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:46.403444 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:46.403447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:46.404136 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:46.404154 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:46.404634 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:46.404638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:46.404739 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:46.404791 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:24:46.405139 (+ 348us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:24:46.405146 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 17:24:46.405202 (+ 56us) client_negotiation.cc:240] Negotiation successful
0923 17:24:46.405328 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":131,"threads_started":1}
I20260923 17:24:46.427688 13921 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:24:46.427872 13921 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:24:46.430166 13921 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a52102167344208bc65d3c5369023c1: Sending DeleteTablet for 3 replicas of tablet 62f5086166d14d2d977885cecb9d359d
I20260923 17:24:46.430934 14199 tablet_service.cc:1514] Processing DeleteTablet for tablet 62f5086166d14d2d977885cecb9d359d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:24:46 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:33552
I20260923 17:24:46.430934 14046 tablet_service.cc:1514] Processing DeleteTablet for tablet 62f5086166d14d2d977885cecb9d359d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:24:46 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:32782
I20260923 17:24:46.431034 14339 tablet_service.cc:1514] Processing DeleteTablet for tablet 62f5086166d14d2d977885cecb9d359d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:24:46 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:55848
I20260923 17:24:46.431131 14507 tablet_replica.cc:330] T 62f5086166d14d2d977885cecb9d359d P 0cda85198f024d738628555ffa00ac19: stopping tablet replica
I20260923 17:24:46.431234 14508 tablet_replica.cc:330] T 62f5086166d14d2d977885cecb9d359d P 2af2232dfff944a9aceafbbd871bdd14: stopping tablet replica
I20260923 17:24:46.431249 14506 tablet_replica.cc:330] T 62f5086166d14d2d977885cecb9d359d P 696b0b38fed74ae1ba7e427b3c138be5: stopping tablet replica
I20260923 17:24:46.431286 14507 raft_consensus.cc:2229] T 62f5086166d14d2d977885cecb9d359d P 0cda85198f024d738628555ffa00ac19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:24:46.431337 14508 raft_consensus.cc:2229] T 62f5086166d14d2d977885cecb9d359d P 2af2232dfff944a9aceafbbd871bdd14 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:24:46.431370 14506 raft_consensus.cc:2229] T 62f5086166d14d2d977885cecb9d359d P 696b0b38fed74ae1ba7e427b3c138be5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:24:46.431406 14507 raft_consensus.cc:2258] T 62f5086166d14d2d977885cecb9d359d P 0cda85198f024d738628555ffa00ac19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:24:46.431492 14506 raft_consensus.cc:2258] T 62f5086166d14d2d977885cecb9d359d P 696b0b38fed74ae1ba7e427b3c138be5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:24:46.431634 14508 raft_consensus.cc:2258] T 62f5086166d14d2d977885cecb9d359d P 2af2232dfff944a9aceafbbd871bdd14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:24:46.432260 14508 ts_tablet_manager.cc:1905] T 62f5086166d14d2d977885cecb9d359d P 2af2232dfff944a9aceafbbd871bdd14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:24:46.432242 14507 ts_tablet_manager.cc:1905] T 62f5086166d14d2d977885cecb9d359d P 0cda85198f024d738628555ffa00ac19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:24:46.432415 14506 ts_tablet_manager.cc:1905] T 62f5086166d14d2d977885cecb9d359d P 696b0b38fed74ae1ba7e427b3c138be5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:24:46.435191 14507 ts_tablet_manager.cc:1918] T 62f5086166d14d2d977885cecb9d359d P 0cda85198f024d738628555ffa00ac19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:24:46.435191 14508 ts_tablet_manager.cc:1918] T 62f5086166d14d2d977885cecb9d359d P 2af2232dfff944a9aceafbbd871bdd14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:24:46.435195 14506 ts_tablet_manager.cc:1918] T 62f5086166d14d2d977885cecb9d359d P 696b0b38fed74ae1ba7e427b3c138be5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:24:46.435266 14506 log.cc:1196] T 62f5086166d14d2d977885cecb9d359d P 696b0b38fed74ae1ba7e427b3c138be5: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-0/wal/wals/62f5086166d14d2d977885cecb9d359d
I20260923 17:24:46.435267 14507 log.cc:1196] T 62f5086166d14d2d977885cecb9d359d P 0cda85198f024d738628555ffa00ac19: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-1/wal/wals/62f5086166d14d2d977885cecb9d359d
I20260923 17:24:46.435267 14508 log.cc:1196] T 62f5086166d14d2d977885cecb9d359d P 2af2232dfff944a9aceafbbd871bdd14: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-2/wal/wals/62f5086166d14d2d977885cecb9d359d
I20260923 17:24:46.435550 14507 ts_tablet_manager.cc:1939] T 62f5086166d14d2d977885cecb9d359d P 0cda85198f024d738628555ffa00ac19: Deleting consensus metadata
I20260923 17:24:46.435549 14508 ts_tablet_manager.cc:1939] T 62f5086166d14d2d977885cecb9d359d P 2af2232dfff944a9aceafbbd871bdd14: Deleting consensus metadata
I20260923 17:24:46.435556 14506 ts_tablet_manager.cc:1939] T 62f5086166d14d2d977885cecb9d359d P 696b0b38fed74ae1ba7e427b3c138be5: Deleting consensus metadata
I20260923 17:24:46.436573 13921 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260923 17:24:46.436843 13907 catalog_manager.cc:4841] TS 2af2232dfff944a9aceafbbd871bdd14 (127.11.229.3:33357): tablet 62f5086166d14d2d977885cecb9d359d (table test-table [id=1f1b5cf787cf4aabb779c5551794ffb0]) successfully deleted
W20260923 17:24:46.437098 13921 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:24:46.437439 13907 catalog_manager.cc:4841] TS 0cda85198f024d738628555ffa00ac19 (127.11.229.2:39963): tablet 62f5086166d14d2d977885cecb9d359d (table test-table [id=1f1b5cf787cf4aabb779c5551794ffb0]) successfully deleted
I20260923 17:24:46.438059 13907 catalog_manager.cc:4841] TS 696b0b38fed74ae1ba7e427b3c138be5 (127.11.229.1:33493): tablet 62f5086166d14d2d977885cecb9d359d (table test-table [id=1f1b5cf787cf4aabb779c5551794ffb0]) successfully deleted
I20260923 17:24:46.442080 14046 tablet_service.cc:1467] Processing CreateTablet for tablet c121241865bd42b6a329c95f4ff513b1 (DEFAULT_TABLE table=test-table [id=4f550adcebd647cbb2c06205526d8bd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:24:46.442229 14199 tablet_service.cc:1467] Processing CreateTablet for tablet c121241865bd42b6a329c95f4ff513b1 (DEFAULT_TABLE table=test-table [id=4f550adcebd647cbb2c06205526d8bd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:24:46.442395 14046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c121241865bd42b6a329c95f4ff513b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:46.442502 14199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c121241865bd42b6a329c95f4ff513b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:46.444851 14518 tablet_bootstrap.cc:492] T c121241865bd42b6a329c95f4ff513b1 P 696b0b38fed74ae1ba7e427b3c138be5: Bootstrap starting.
I20260923 17:24:46.445422 14517 tablet_bootstrap.cc:492] T c121241865bd42b6a329c95f4ff513b1 P 0cda85198f024d738628555ffa00ac19: Bootstrap starting.
I20260923 17:24:46.445760 14518 tablet_bootstrap.cc:654] T c121241865bd42b6a329c95f4ff513b1 P 696b0b38fed74ae1ba7e427b3c138be5: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:46.446350 14517 tablet_bootstrap.cc:654] T c121241865bd42b6a329c95f4ff513b1 P 0cda85198f024d738628555ffa00ac19: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:46.446935 14518 tablet_bootstrap.cc:492] T c121241865bd42b6a329c95f4ff513b1 P 696b0b38fed74ae1ba7e427b3c138be5: No bootstrap required, opened a new log
I20260923 17:24:46.447011 14518 ts_tablet_manager.cc:1397] T c121241865bd42b6a329c95f4ff513b1 P 696b0b38fed74ae1ba7e427b3c138be5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 17:24:46.447086 14339 tablet_service.cc:1467] Processing CreateTablet for tablet c121241865bd42b6a329c95f4ff513b1 (DEFAULT_TABLE table=test-table [id=4f550adcebd647cbb2c06205526d8bd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:24:46.447330 14339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c121241865bd42b6a329c95f4ff513b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:46.447378 14518 raft_consensus.cc:348] T c121241865bd42b6a329c95f4ff513b1 P 696b0b38fed74ae1ba7e427b3c138be5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } }
I20260923 17:24:46.447497 14518 raft_consensus.cc:374] T c121241865bd42b6a329c95f4ff513b1 P 696b0b38fed74ae1ba7e427b3c138be5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:46.447531 14518 raft_consensus.cc:729] T c121241865bd42b6a329c95f4ff513b1 P 696b0b38fed74ae1ba7e427b3c138be5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696b0b38fed74ae1ba7e427b3c138be5, State: Initialized, Role: FOLLOWER
I20260923 17:24:46.447659 14518 consensus_queue.cc:260] T c121241865bd42b6a329c95f4ff513b1 P 696b0b38fed74ae1ba7e427b3c138be5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } }
I20260923 17:24:46.447908 14518 ts_tablet_manager.cc:1428] T c121241865bd42b6a329c95f4ff513b1 P 696b0b38fed74ae1ba7e427b3c138be5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 17:24:46.449699 14519 tablet_bootstrap.cc:492] T c121241865bd42b6a329c95f4ff513b1 P 2af2232dfff944a9aceafbbd871bdd14: Bootstrap starting.
I20260923 17:24:46.450630 14519 tablet_bootstrap.cc:654] T c121241865bd42b6a329c95f4ff513b1 P 2af2232dfff944a9aceafbbd871bdd14: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:46.450814 14517 tablet_bootstrap.cc:492] T c121241865bd42b6a329c95f4ff513b1 P 0cda85198f024d738628555ffa00ac19: No bootstrap required, opened a new log
I20260923 17:24:46.450915 14517 ts_tablet_manager.cc:1397] T c121241865bd42b6a329c95f4ff513b1 P 0cda85198f024d738628555ffa00ac19: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260923 17:24:46.451380 14517 raft_consensus.cc:348] T c121241865bd42b6a329c95f4ff513b1 P 0cda85198f024d738628555ffa00ac19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } }
I20260923 17:24:46.451471 14517 raft_consensus.cc:374] T c121241865bd42b6a329c95f4ff513b1 P 0cda85198f024d738628555ffa00ac19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:46.451491 14517 raft_consensus.cc:729] T c121241865bd42b6a329c95f4ff513b1 P 0cda85198f024d738628555ffa00ac19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cda85198f024d738628555ffa00ac19, State: Initialized, Role: FOLLOWER
I20260923 17:24:46.451591 14517 consensus_queue.cc:260] T c121241865bd42b6a329c95f4ff513b1 P 0cda85198f024d738628555ffa00ac19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } }
I20260923 17:24:46.451844 14517 ts_tablet_manager.cc:1428] T c121241865bd42b6a329c95f4ff513b1 P 0cda85198f024d738628555ffa00ac19: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 17:24:46.452855 14519 tablet_bootstrap.cc:492] T c121241865bd42b6a329c95f4ff513b1 P 2af2232dfff944a9aceafbbd871bdd14: No bootstrap required, opened a new log
I20260923 17:24:46.452962 14519 ts_tablet_manager.cc:1397] T c121241865bd42b6a329c95f4ff513b1 P 2af2232dfff944a9aceafbbd871bdd14: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 17:24:46.453411 14519 raft_consensus.cc:348] T c121241865bd42b6a329c95f4ff513b1 P 2af2232dfff944a9aceafbbd871bdd14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } }
I20260923 17:24:46.453544 14519 raft_consensus.cc:374] T c121241865bd42b6a329c95f4ff513b1 P 2af2232dfff944a9aceafbbd871bdd14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:46.453576 14519 raft_consensus.cc:729] T c121241865bd42b6a329c95f4ff513b1 P 2af2232dfff944a9aceafbbd871bdd14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2af2232dfff944a9aceafbbd871bdd14, State: Initialized, Role: FOLLOWER
I20260923 17:24:46.453693 14519 consensus_queue.cc:260] T c121241865bd42b6a329c95f4ff513b1 P 2af2232dfff944a9aceafbbd871bdd14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } }
I20260923 17:24:46.453959 14519 ts_tablet_manager.cc:1428] T c121241865bd42b6a329c95f4ff513b1 P 2af2232dfff944a9aceafbbd871bdd14: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 17:24:46.479244 14461 raft_consensus.cc:482] T c121241865bd42b6a329c95f4ff513b1 P 696b0b38fed74ae1ba7e427b3c138be5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:24:46.479396 14461 raft_consensus.cc:504] T c121241865bd42b6a329c95f4ff513b1 P 696b0b38fed74ae1ba7e427b3c138be5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } }
I20260923 17:24:46.479879 14461 leader_election.cc:288] T c121241865bd42b6a329c95f4ff513b1 P 696b0b38fed74ae1ba7e427b3c138be5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cda85198f024d738628555ffa00ac19 (127.11.229.2:39963), 2af2232dfff944a9aceafbbd871bdd14 (127.11.229.3:33357)
I20260923 17:24:46.480346 14218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c121241865bd42b6a329c95f4ff513b1" candidate_uuid: "696b0b38fed74ae1ba7e427b3c138be5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cda85198f024d738628555ffa00ac19" is_pre_election: true
I20260923 17:24:46.480530 14218 raft_consensus.cc:2454] T c121241865bd42b6a329c95f4ff513b1 P 0cda85198f024d738628555ffa00ac19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696b0b38fed74ae1ba7e427b3c138be5 in term 0.
I20260923 17:24:46.480584 14357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c121241865bd42b6a329c95f4ff513b1" candidate_uuid: "696b0b38fed74ae1ba7e427b3c138be5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2af2232dfff944a9aceafbbd871bdd14" is_pre_election: true
I20260923 17:24:46.480756 14357 raft_consensus.cc:2454] T c121241865bd42b6a329c95f4ff513b1 P 2af2232dfff944a9aceafbbd871bdd14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696b0b38fed74ae1ba7e427b3c138be5 in term 0.
I20260923 17:24:46.480906 14011 leader_election.cc:302] T c121241865bd42b6a329c95f4ff513b1 P 696b0b38fed74ae1ba7e427b3c138be5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cda85198f024d738628555ffa00ac19, 696b0b38fed74ae1ba7e427b3c138be5; no voters:
I20260923 17:24:46.481091 14461 raft_consensus.cc:2781] T c121241865bd42b6a329c95f4ff513b1 P 696b0b38fed74ae1ba7e427b3c138be5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:24:46.481160 14461 raft_consensus.cc:482] T c121241865bd42b6a329c95f4ff513b1 P 696b0b38fed74ae1ba7e427b3c138be5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:24:46.481199 14461 raft_consensus.cc:3037] T c121241865bd42b6a329c95f4ff513b1 P 696b0b38fed74ae1ba7e427b3c138be5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:46.482518 14461 raft_consensus.cc:504] T c121241865bd42b6a329c95f4ff513b1 P 696b0b38fed74ae1ba7e427b3c138be5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } }
I20260923 17:24:46.482884 14461 leader_election.cc:288] T c121241865bd42b6a329c95f4ff513b1 P 696b0b38fed74ae1ba7e427b3c138be5 [CANDIDATE]: Term 1 election: Requested vote from peers 0cda85198f024d738628555ffa00ac19 (127.11.229.2:39963), 2af2232dfff944a9aceafbbd871bdd14 (127.11.229.3:33357)
I20260923 17:24:46.483367 14218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c121241865bd42b6a329c95f4ff513b1" candidate_uuid: "696b0b38fed74ae1ba7e427b3c138be5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cda85198f024d738628555ffa00ac19"
I20260923 17:24:46.483464 14357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c121241865bd42b6a329c95f4ff513b1" candidate_uuid: "696b0b38fed74ae1ba7e427b3c138be5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2af2232dfff944a9aceafbbd871bdd14"
I20260923 17:24:46.483507 14218 raft_consensus.cc:3037] T c121241865bd42b6a329c95f4ff513b1 P 0cda85198f024d738628555ffa00ac19 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:46.483592 14357 raft_consensus.cc:3037] T c121241865bd42b6a329c95f4ff513b1 P 2af2232dfff944a9aceafbbd871bdd14 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:46.484848 14357 raft_consensus.cc:2454] T c121241865bd42b6a329c95f4ff513b1 P 2af2232dfff944a9aceafbbd871bdd14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696b0b38fed74ae1ba7e427b3c138be5 in term 1.
I20260923 17:24:46.484846 14218 raft_consensus.cc:2454] T c121241865bd42b6a329c95f4ff513b1 P 0cda85198f024d738628555ffa00ac19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696b0b38fed74ae1ba7e427b3c138be5 in term 1.
I20260923 17:24:46.485180 14011 leader_election.cc:302] T c121241865bd42b6a329c95f4ff513b1 P 696b0b38fed74ae1ba7e427b3c138be5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cda85198f024d738628555ffa00ac19, 696b0b38fed74ae1ba7e427b3c138be5; no voters:
I20260923 17:24:46.485533 14461 raft_consensus.cc:2781] T c121241865bd42b6a329c95f4ff513b1 P 696b0b38fed74ae1ba7e427b3c138be5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:24:46.485671 14461 raft_consensus.cc:686] T c121241865bd42b6a329c95f4ff513b1 P 696b0b38fed74ae1ba7e427b3c138be5 [term 1 LEADER]: Becoming Leader. State: Replica: 696b0b38fed74ae1ba7e427b3c138be5, State: Running, Role: LEADER
I20260923 17:24:46.485810 14461 consensus_queue.cc:237] T c121241865bd42b6a329c95f4ff513b1 P 696b0b38fed74ae1ba7e427b3c138be5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } }
I20260923 17:24:46.487576 13922 catalog_manager.cc:5495] T c121241865bd42b6a329c95f4ff513b1 P 696b0b38fed74ae1ba7e427b3c138be5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696b0b38fed74ae1ba7e427b3c138be5 (127.11.229.1). New cstate: current_term: 1 leader_uuid: "696b0b38fed74ae1ba7e427b3c138be5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696b0b38fed74ae1ba7e427b3c138be5" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 33493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 } health_report { overall_health: UNKNOWN } } }
I20260923 17:24:46.510593 14528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:46.507024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50602 (local address 127.11.229.1:33493)
0923 17:24:46.507140 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:46.507145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:46.507182 (+ 37us) server_negotiation.cc:407] Connection header received
0923 17:24:46.507338 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:46.507341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:46.507393 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:46.507483 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:24:46.507713 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:46.508502 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:46.509458 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:46.509810 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:46.509917 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:46.510139 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:24:46.510226 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:24:46.510297 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:46.510380 (+ 83us) server_negotiation.cc:299] Negotiation successful
0923 17:24:46.510433 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 17:24:46.518054 14357 raft_consensus.cc:1261] T c121241865bd42b6a329c95f4ff513b1 P 2af2232dfff944a9aceafbbd871bdd14 [term 1 FOLLOWER]: Refusing update from remote peer 696b0b38fed74ae1ba7e427b3c138be5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:24:46.518307 14218 raft_consensus.cc:1261] T c121241865bd42b6a329c95f4ff513b1 P 0cda85198f024d738628555ffa00ac19 [term 1 FOLLOWER]: Refusing update from remote peer 696b0b38fed74ae1ba7e427b3c138be5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:24:46.518771 14461 consensus_queue.cc:1035] T c121241865bd42b6a329c95f4ff513b1 P 696b0b38fed74ae1ba7e427b3c138be5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cda85198f024d738628555ffa00ac19" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:24:46.519011 14500 consensus_queue.cc:1035] T c121241865bd42b6a329c95f4ff513b1 P 696b0b38fed74ae1ba7e427b3c138be5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2af2232dfff944a9aceafbbd871bdd14" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 17:24:46.533722 14487 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 17:24:48.092831 13918 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:24:48.093031 13918 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:24:48.095562 13918 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a52102167344208bc65d3c5369023c1: Sending DeleteTablet for 3 replicas of tablet c121241865bd42b6a329c95f4ff513b1
I20260923 17:24:48.096087 14046 tablet_service.cc:1514] Processing DeleteTablet for tablet c121241865bd42b6a329c95f4ff513b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:24:48 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:32782
I20260923 17:24:48.096388 14199 tablet_service.cc:1514] Processing DeleteTablet for tablet c121241865bd42b6a329c95f4ff513b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:24:48 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:33552
I20260923 17:24:48.096410 14339 tablet_service.cc:1514] Processing DeleteTablet for tablet c121241865bd42b6a329c95f4ff513b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:24:48 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:55848
I20260923 17:24:48.096474 14568 tablet_replica.cc:330] T c121241865bd42b6a329c95f4ff513b1 P 696b0b38fed74ae1ba7e427b3c138be5: stopping tablet replica
I20260923 17:24:48.096570 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 13994
I20260923 17:24:48.097939 14570 tablet_replica.cc:330] T c121241865bd42b6a329c95f4ff513b1 P 2af2232dfff944a9aceafbbd871bdd14: stopping tablet replica
I20260923 17:24:48.098132 14570 raft_consensus.cc:2229] T c121241865bd42b6a329c95f4ff513b1 P 2af2232dfff944a9aceafbbd871bdd14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:24:48.098477 14570 raft_consensus.cc:2258] T c121241865bd42b6a329c95f4ff513b1 P 2af2232dfff944a9aceafbbd871bdd14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:24:48.099315 14570 ts_tablet_manager.cc:1905] T c121241865bd42b6a329c95f4ff513b1 P 2af2232dfff944a9aceafbbd871bdd14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:24:48.101451 14569 tablet_replica.cc:330] T c121241865bd42b6a329c95f4ff513b1 P 0cda85198f024d738628555ffa00ac19: stopping tablet replica
I20260923 17:24:48.101606 14569 raft_consensus.cc:2229] T c121241865bd42b6a329c95f4ff513b1 P 0cda85198f024d738628555ffa00ac19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:24:48.101744 14569 raft_consensus.cc:2258] T c121241865bd42b6a329c95f4ff513b1 P 0cda85198f024d738628555ffa00ac19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:24:48.102351 14569 ts_tablet_manager.cc:1905] T c121241865bd42b6a329c95f4ff513b1 P 0cda85198f024d738628555ffa00ac19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:24:48.102926 14570 ts_tablet_manager.cc:1918] T c121241865bd42b6a329c95f4ff513b1 P 2af2232dfff944a9aceafbbd871bdd14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 17:24:48.103009 14570 log.cc:1196] T c121241865bd42b6a329c95f4ff513b1 P 2af2232dfff944a9aceafbbd871bdd14: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-2/wal/wals/c121241865bd42b6a329c95f4ff513b1
I20260923 17:24:48.103279 14570 ts_tablet_manager.cc:1939] T c121241865bd42b6a329c95f4ff513b1 P 2af2232dfff944a9aceafbbd871bdd14: Deleting consensus metadata
I20260923 17:24:48.104071 13907 catalog_manager.cc:4841] TS 2af2232dfff944a9aceafbbd871bdd14 (127.11.229.3:33357): tablet c121241865bd42b6a329c95f4ff513b1 (table test-table [id=4f550adcebd647cbb2c06205526d8bd4]) successfully deleted
I20260923 17:24:48.104669 14569 ts_tablet_manager.cc:1918] T c121241865bd42b6a329c95f4ff513b1 P 0cda85198f024d738628555ffa00ac19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 17:24:48.104763 14569 log.cc:1196] T c121241865bd42b6a329c95f4ff513b1 P 0cda85198f024d738628555ffa00ac19: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184271597484-12180-0/minicluster-data/ts-1/wal/wals/c121241865bd42b6a329c95f4ff513b1
I20260923 17:24:48.105017 14569 ts_tablet_manager.cc:1939] T c121241865bd42b6a329c95f4ff513b1 P 0cda85198f024d738628555ffa00ac19: Deleting consensus metadata
I20260923 17:24:48.105720 13907 catalog_manager.cc:4841] TS 0cda85198f024d738628555ffa00ac19 (127.11.229.2:39963): tablet c121241865bd42b6a329c95f4ff513b1 (table test-table [id=4f550adcebd647cbb2c06205526d8bd4]) successfully deleted
W20260923 17:24:48.106503 13907 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.229.1:33493 (error 108)
I20260923 17:24:48.106747 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 14134
I20260923 17:24:48.108181 14572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:48.107770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:33493 (local address 127.0.0.1:50616)
0923 17:24:48.108012 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:48.108095 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.1:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":94,"threads_started":1}
W20260923 17:24:48.108335 13907 catalog_manager.cc:4584] TS 696b0b38fed74ae1ba7e427b3c138be5 (127.11.229.1:33493): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c121241865bd42b6a329c95f4ff513b1: Network error: Client connection negotiation failed: client connection to 127.11.229.1:33493: connect: Connection refused (error 111)
I20260923 17:24:48.114804 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 14274
I20260923 17:24:48.121986 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 13890
2026-09-23T17:24:48Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6478 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:24:48 dist-test-slave-ptl9 krb5kdc[14575](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:24:48 dist-test-slave-ptl9 krb5kdc[14575](info): set up 2 sockets
Sep 23 17:24:48 dist-test-slave-ptl9 krb5kdc[14575](info): commencing operation
krb5kdc: starting...
W20260923 17:24:49.201705 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:24:49 dist-test-slave-ptl9 krb5kdc[14575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:24:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:24:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.229.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
I20260923 17:24:49.328303 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:46477
--webserver_interface=127.11.229.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:46477
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:49.440661 14588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:49.441015 14588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:49.441097 14588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:49.445369 14588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:24:49.445461 14588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:49.445497 14588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:49.445557 14588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:24:49.445613 14588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:24:49.451105 14588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:46477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:46477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:49.452312 14588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:49.453449 14588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:49.462299 14594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:49.462327 14593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:49.462347 14588 server_base.cc:1034] running on GCE node
W20260923 17:24:49.462360 14596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:49.463106 14588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:49.464340 14588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:49.465533 14588 hybrid_clock.cc:648] HybridClock initialized: now 1790184289465516 us; error 36 us; skew 500 ppm
Sep 23 17:24:49 dist-test-slave-ptl9 krb5kdc[14575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184289, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:49.468328 14588 init.cc:377] Logged in from keytab as kudu/127.11.229.62@KRBTEST.COM (short username kudu)
I20260923 17:24:49.469249 14588 webserver.cc:460] Webserver started at http://127.11.229.62:45699/ using document root <none> and password file <none>
I20260923 17:24:49.469856 14588 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:49.469916 14588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:49.470079 14588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:49.472182 14588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "6c88cde38f1742e7a3d54cee0728893d"
format_stamp: "Formatted at 2026-09-23 17:24:49 on dist-test-slave-ptl9"
server_key: "04a22a2c9916c996359ade57bbd24cf3"
server_key_iv: "1bfc47164884ccd465dabe76bf9c1aed"
server_key_version: "encryptionkey@0"
I20260923 17:24:49.472733 14588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "6c88cde38f1742e7a3d54cee0728893d"
format_stamp: "Formatted at 2026-09-23 17:24:49 on dist-test-slave-ptl9"
server_key: "04a22a2c9916c996359ade57bbd24cf3"
server_key_iv: "1bfc47164884ccd465dabe76bf9c1aed"
server_key_version: "encryptionkey@0"
I20260923 17:24:49.476656 14588 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260923 17:24:49.479204 14603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:49.480600 14588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260923 17:24:49.480734 14588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "6c88cde38f1742e7a3d54cee0728893d"
format_stamp: "Formatted at 2026-09-23 17:24:49 on dist-test-slave-ptl9"
server_key: "04a22a2c9916c996359ade57bbd24cf3"
server_key_iv: "1bfc47164884ccd465dabe76bf9c1aed"
server_key_version: "encryptionkey@0"
I20260923 17:24:49.480847 14588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:49.492188 14588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:49.495270 14588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:49.495493 14588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:49.502920 14588 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:46477
I20260923 17:24:49.502942 14655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:46477 every 8 connection(s)
I20260923 17:24:49.503932 14588 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:24:49.504648 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 14588
I20260923 17:24:49.504761 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:24:49.505044 12180 external_mini_cluster.cc:1428] Setting key 2e880006b33ce3bc1fb0f47d91f866d9
I20260923 17:24:49.507251 14656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:49.514173 14656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c88cde38f1742e7a3d54cee0728893d: Bootstrap starting.
Sep 23 17:24:49 dist-test-slave-ptl9 krb5kdc[14575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:24:49.516737 14656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c88cde38f1742e7a3d54cee0728893d: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:49.517555 14656 log.cc:824] T 00000000000000000000000000000000 P 6c88cde38f1742e7a3d54cee0728893d: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:49.519562 14656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c88cde38f1742e7a3d54cee0728893d: No bootstrap required, opened a new log
I20260923 17:24:49.519737 14659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.506487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34918 (local address 127.11.229.62:46477)
0923 17:24:49.506910 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:49.506935 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:49.506971 (+ 36us) server_negotiation.cc:407] Connection header received
0923 17:24:49.507728 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:49.507746 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:49.508099 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:49.508385 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:49.509348 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.510349 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.511518 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.511995 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.512110 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:49.515033 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:49.515038 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:49.515052 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:49.515080 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:49.516798 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:49.517313 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:49.517319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:49.517324 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:49.517436 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:49.517769 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:49.517771 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:49.517772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:49.518168 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:49.518380 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:49.518693 (+ 313us) server_negotiation.cc:299] Negotiation successful
0923 17:24:49.518982 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":142,"threads_started":1}
I20260923 17:24:49.523023 14656 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c88cde38f1742e7a3d54cee0728893d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c88cde38f1742e7a3d54cee0728893d" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46477 } }
I20260923 17:24:49.523278 14656 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c88cde38f1742e7a3d54cee0728893d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:49.523358 14656 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c88cde38f1742e7a3d54cee0728893d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c88cde38f1742e7a3d54cee0728893d, State: Initialized, Role: FOLLOWER
I20260923 17:24:49.523787 14656 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c88cde38f1742e7a3d54cee0728893d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c88cde38f1742e7a3d54cee0728893d" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46477 } }
I20260923 17:24:49.523942 14656 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c88cde38f1742e7a3d54cee0728893d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:24:49.524030 14656 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c88cde38f1742e7a3d54cee0728893d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:24:49.524166 14656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c88cde38f1742e7a3d54cee0728893d [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:49.525010 14656 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c88cde38f1742e7a3d54cee0728893d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c88cde38f1742e7a3d54cee0728893d" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46477 } }
I20260923 17:24:49.525388 14656 leader_election.cc:302] T 00000000000000000000000000000000 P 6c88cde38f1742e7a3d54cee0728893d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c88cde38f1742e7a3d54cee0728893d; no voters:
I20260923 17:24:49.525763 14656 leader_election.cc:288] T 00000000000000000000000000000000 P 6c88cde38f1742e7a3d54cee0728893d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:24:49.525867 14661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c88cde38f1742e7a3d54cee0728893d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:24:49.526090 14661 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c88cde38f1742e7a3d54cee0728893d [term 1 LEADER]: Becoming Leader. State: Replica: 6c88cde38f1742e7a3d54cee0728893d, State: Running, Role: LEADER
I20260923 17:24:49.526430 14661 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c88cde38f1742e7a3d54cee0728893d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c88cde38f1742e7a3d54cee0728893d" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46477 } }
I20260923 17:24:49.527019 14656 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c88cde38f1742e7a3d54cee0728893d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:24:49.528296 14662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c88cde38f1742e7a3d54cee0728893d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c88cde38f1742e7a3d54cee0728893d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c88cde38f1742e7a3d54cee0728893d" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46477 } } }
I20260923 17:24:49.528420 14662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c88cde38f1742e7a3d54cee0728893d [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:49.528657 14663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c88cde38f1742e7a3d54cee0728893d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c88cde38f1742e7a3d54cee0728893d. Latest consensus state: current_term: 1 leader_uuid: "6c88cde38f1742e7a3d54cee0728893d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c88cde38f1742e7a3d54cee0728893d" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46477 } } }
I20260923 17:24:49.528748 14663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c88cde38f1742e7a3d54cee0728893d [sys.catalog]: This master's current role is: LEADER
W20260923 17:24:49.532207 14676 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6c88cde38f1742e7a3d54cee0728893d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:24:49.532291 14676 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:24:49.532351 14670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:24:49.533320 14670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:24:49.539155 14670 catalog_manager.cc:1347] Generated new cluster ID: 861db037fd844aadafac05c3ce06826f
I20260923 17:24:49.539232 14670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:24:49.558166 14670 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:24:49.559504 14670 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:24:49.569556 14670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c88cde38f1742e7a3d54cee0728893d: Generated new TSK 0
I20260923 17:24:49.570472 14670 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.229.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab.
Entry for principal kudu/127.11.229.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab.
I20260923 17:24:49.600420 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.1:0
--local_ip_for_outbound_sockets=127.11.229.1
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:46477
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:49.715236 14682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:49.715517 14682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:49.715559 14682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:49.719710 14682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:49.719779 14682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:49.719837 14682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.1
I20260923 17:24:49.725244 14682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:46477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.1
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:49.726454 14682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:49.727624 14682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:49.737871 14688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:49.737891 14687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:49.737870 14690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:49.738080 14682 server_base.cc:1034] running on GCE node
I20260923 17:24:49.738689 14682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:49.739391 14682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:49.740584 14682 hybrid_clock.cc:648] HybridClock initialized: now 1790184289740569 us; error 29 us; skew 500 ppm
Sep 23 17:24:49 dist-test-slave-ptl9 krb5kdc[14575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184289, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:49.743580 14682 init.cc:377] Logged in from keytab as kudu/127.11.229.1@KRBTEST.COM (short username kudu)
I20260923 17:24:49.744616 14682 webserver.cc:460] Webserver started at http://127.11.229.1:38151/ using document root <none> and password file <none>
I20260923 17:24:49.745368 14682 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:49.745467 14682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:49.745688 14682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:49.747959 14682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-0/data/instance:
uuid: "676586a1f62f436db5d68dd6989b9831"
format_stamp: "Formatted at 2026-09-23 17:24:49 on dist-test-slave-ptl9"
server_key: "9c013a26faf7ecd60ba9000589f9a85b"
server_key_iv: "053e46e2500ec993b68fde40ea7cf4bc"
server_key_version: "encryptionkey@0"
I20260923 17:24:49.748559 14682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance:
uuid: "676586a1f62f436db5d68dd6989b9831"
format_stamp: "Formatted at 2026-09-23 17:24:49 on dist-test-slave-ptl9"
server_key: "9c013a26faf7ecd60ba9000589f9a85b"
server_key_iv: "053e46e2500ec993b68fde40ea7cf4bc"
server_key_version: "encryptionkey@0"
I20260923 17:24:49.752496 14682 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 17:24:49.755220 14697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:49.756592 14682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260923 17:24:49.756747 14682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-0/wal
uuid: "676586a1f62f436db5d68dd6989b9831"
format_stamp: "Formatted at 2026-09-23 17:24:49 on dist-test-slave-ptl9"
server_key: "9c013a26faf7ecd60ba9000589f9a85b"
server_key_iv: "053e46e2500ec993b68fde40ea7cf4bc"
server_key_version: "encryptionkey@0"
I20260923 17:24:49.756871 14682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:49.771806 14682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:49.775030 14682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:49.775251 14682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:49.775919 14682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:24:49.776862 14682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:24:49.776932 14682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:49.777007 14682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:24:49.777050 14682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:49.786567 14682 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.1:45333
I20260923 17:24:49.786581 14810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.1:45333 every 8 connection(s)
I20260923 17:24:49.787611 14682 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
I20260923 17:24:49.797092 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 14682
I20260923 17:24:49.797224 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance
Sep 23 17:24:49 dist-test-slave-ptl9 krb5kdc[14575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184289, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:24:49.797569 12180 external_mini_cluster.cc:1428] Setting key b62b100cd0ddc6fc21832a2fa3d38271
I20260923 17:24:49.801962 14659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.789695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:43769 (local address 127.11.229.62:46477)
0923 17:24:49.789838 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:49.789841 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:49.790932 (+ 1091us) server_negotiation.cc:407] Connection header received
0923 17:24:49.791868 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:49.791871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:49.791920 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:49.791993 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:49.793190 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.793788 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.794995 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.795270 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.795354 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:49.798138 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:49.798142 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:49.798145 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:49.798181 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:49.799768 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:49.800325 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:49.800333 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:49.800334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:49.800411 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:49.800764 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:49.800769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:49.800771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:49.800969 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:49.801078 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:49.801664 (+ 586us) server_negotiation.cc:299] Negotiation successful
0923 17:24:49.801799 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 17:24:49.803192 14813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:49.790063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46477 (local address 127.11.229.1:43769)
0923 17:24:49.790796 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:49.790840 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:49.791666 (+ 826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:49.792138 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:49.792145 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:49.792510 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:49.793031 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:49.793053 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.793921 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.793924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:49.794857 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:49.794867 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:49.795414 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:49.795417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:49.795580 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:49.796254 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:49.796296 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:49.797952 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:49.799903 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:49.799909 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:49.799927 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:49.800209 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:49.800505 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:49.800508 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:49.800510 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:49.800664 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:49.801083 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:49.801092 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:49.801420 (+ 328us) client_negotiation.cc:769] Sending connection context
0923 17:24:49.801624 (+ 204us) client_negotiation.cc:240] Negotiation successful
0923 17:24:49.801973 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":120,"threads_started":1}
I20260923 17:24:49.804610 14811 heartbeater.cc:347] Connected to a master server at 127.11.229.62:46477
I20260923 17:24:49.804898 14811 heartbeater.cc:464] Registering TS with master...
I20260923 17:24:49.805579 14811 heartbeater.cc:511] Master 127.11.229.62:46477 requested a full tablet report, sending...
I20260923 17:24:49.807298 14620 ts_manager.cc:194] Registered new tserver with Master: 676586a1f62f436db5d68dd6989b9831 (127.11.229.1:45333)
WARNING: no policy specified for kudu/127.11.229.2@KRBTEST.COM; defaulting to no policy
I20260923 17:24:49.808642 14620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.1@KRBTEST.COM'} at 127.11.229.1:43769
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab.
Entry for principal kudu/127.11.229.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab.
I20260923 17:24:49.828142 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.2:0
--local_ip_for_outbound_sockets=127.11.229.2
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:46477
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:49.941602 14816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:49.941859 14816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:49.941905 14816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:49.946025 14816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:49.946084 14816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:49.946143 14816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.2
I20260923 17:24:49.951388 14816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:46477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.2
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:49.952373 14816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:49.953418 14816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:49.963835 14822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:49.963783 14821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:49.963858 14816 server_base.cc:1034] running on GCE node
W20260923 17:24:49.963790 14824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:49.964514 14816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:49.965302 14816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:49.966516 14816 hybrid_clock.cc:648] HybridClock initialized: now 1790184289966497 us; error 27 us; skew 500 ppm
Sep 23 17:24:49 dist-test-slave-ptl9 krb5kdc[14575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184289, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:49.969475 14816 init.cc:377] Logged in from keytab as kudu/127.11.229.2@KRBTEST.COM (short username kudu)
I20260923 17:24:49.970358 14816 webserver.cc:460] Webserver started at http://127.11.229.2:34885/ using document root <none> and password file <none>
I20260923 17:24:49.970937 14816 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:49.971014 14816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:49.971282 14816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:49.973415 14816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-1/data/instance:
uuid: "8737d3c4087a4ebdb80ac78a24716be0"
format_stamp: "Formatted at 2026-09-23 17:24:49 on dist-test-slave-ptl9"
server_key: "a386481afbfc353cf3f8fd5fb3c1b911"
server_key_iv: "72f642f78712d3dcacdb9f106c0a2900"
server_key_version: "encryptionkey@0"
I20260923 17:24:49.973994 14816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance:
uuid: "8737d3c4087a4ebdb80ac78a24716be0"
format_stamp: "Formatted at 2026-09-23 17:24:49 on dist-test-slave-ptl9"
server_key: "a386481afbfc353cf3f8fd5fb3c1b911"
server_key_iv: "72f642f78712d3dcacdb9f106c0a2900"
server_key_version: "encryptionkey@0"
I20260923 17:24:49.977922 14816 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:24:49.980500 14831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:49.981820 14816 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260923 17:24:49.981966 14816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-1/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-1/wal
uuid: "8737d3c4087a4ebdb80ac78a24716be0"
format_stamp: "Formatted at 2026-09-23 17:24:49 on dist-test-slave-ptl9"
server_key: "a386481afbfc353cf3f8fd5fb3c1b911"
server_key_iv: "72f642f78712d3dcacdb9f106c0a2900"
server_key_version: "encryptionkey@0"
I20260923 17:24:49.982087 14816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:49.999521 14816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:50.002595 14816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:50.002820 14816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:50.003465 14816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:24:50.004343 14816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:24:50.004426 14816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:50.004532 14816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:24:50.004587 14816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:50.013521 14816 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.2:36331
I20260923 17:24:50.013543 14944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.2:36331 every 8 connection(s)
I20260923 17:24:50.014485 14816 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
I20260923 17:24:50.014698 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 14816
I20260923 17:24:50.014837 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance
I20260923 17:24:50.015187 12180 external_mini_cluster.cc:1428] Setting key 89ac6230d1d61f16d9d2d77599eb933b
WARNING: no policy specified for kudu/127.11.229.3@KRBTEST.COM; defaulting to no policy
Sep 23 17:24:50 dist-test-slave-ptl9 krb5kdc[14575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184289, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.3@KRBTEST.COM" created.
I20260923 17:24:50.030076 14659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:50.016423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:36431 (local address 127.11.229.62:46477)
0923 17:24:50.016575 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:50.016580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:50.017903 (+ 1323us) server_negotiation.cc:407] Connection header received
0923 17:24:50.019029 (+ 1126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:50.019033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:50.019099 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:50.019204 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:50.020726 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:50.021668 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:50.022855 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:50.023255 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:50.023376 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:50.025909 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:50.025912 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:50.025914 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:50.025938 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:50.027618 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:50.028303 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:50.028310 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:50.028315 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:50.028408 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:50.028797 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:50.028802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:50.028804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:50.028993 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:50.029134 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:50.029780 (+ 646us) server_negotiation.cc:299] Negotiation successful
0923 17:24:50.029900 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 17:24:50.030915 14947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:50.016975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46477 (local address 127.11.229.2:36431)
0923 17:24:50.017756 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:50.017800 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:50.018719 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:50.019429 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:50.019440 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:50.020004 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:50.020559 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:50.020575 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:50.021820 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:50.021823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:50.022719 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:50.022729 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:50.023421 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:50.023424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:50.023569 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:50.024236 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:50.024251 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:50.025740 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:50.027727 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:50.027733 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:50.027749 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:50.028160 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:50.028520 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:50.028524 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:50.028526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:50.028695 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:50.029142 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:50.029152 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:50.029506 (+ 354us) client_negotiation.cc:769] Sending connection context
0923 17:24:50.029738 (+ 232us) client_negotiation.cc:240] Negotiation successful
0923 17:24:50.030065 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":166,"threads_started":1}
I20260923 17:24:50.032265 14945 heartbeater.cc:347] Connected to a master server at 127.11.229.62:46477
I20260923 17:24:50.032518 14945 heartbeater.cc:464] Registering TS with master...
I20260923 17:24:50.033049 14945 heartbeater.cc:511] Master 127.11.229.62:46477 requested a full tablet report, sending...
I20260923 17:24:50.034269 14620 ts_manager.cc:194] Registered new tserver with Master: 8737d3c4087a4ebdb80ac78a24716be0 (127.11.229.2:36331)
I20260923 17:24:50.035099 14620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.2@KRBTEST.COM'} at 127.11.229.2:36431
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab.
Entry for principal kudu/127.11.229.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab.
I20260923 17:24:50.045145 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.3:0
--local_ip_for_outbound_sockets=127.11.229.3
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:46477
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:50.158248 14950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:50.158516 14950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:50.158560 14950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:50.162825 14950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:50.162882 14950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:50.162941 14950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.3
I20260923 17:24:50.168130 14950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:46477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.3
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:50.169159 14950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:50.170169 14950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:50.180224 14955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:50.180224 14958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:50.180225 14956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:50.180477 14950 server_base.cc:1034] running on GCE node
I20260923 17:24:50.180876 14950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:50.181550 14950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:50.182711 14950 hybrid_clock.cc:648] HybridClock initialized: now 1790184290182693 us; error 30 us; skew 500 ppm
Sep 23 17:24:50 dist-test-slave-ptl9 krb5kdc[14575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184290, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:50.185639 14950 init.cc:377] Logged in from keytab as kudu/127.11.229.3@KRBTEST.COM (short username kudu)
I20260923 17:24:50.186618 14950 webserver.cc:460] Webserver started at http://127.11.229.3:45351/ using document root <none> and password file <none>
I20260923 17:24:50.187232 14950 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:50.187294 14950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:50.187522 14950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:50.189646 14950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-2/data/instance:
uuid: "35ea8b120ee44e7bbf4d903c6287433d"
format_stamp: "Formatted at 2026-09-23 17:24:50 on dist-test-slave-ptl9"
server_key: "fd7dd9a9e5918a0b0962f64cf3cbfe4d"
server_key_iv: "cdb6827ef31079074314299b617ce589"
server_key_version: "encryptionkey@0"
I20260923 17:24:50.190207 14950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance:
uuid: "35ea8b120ee44e7bbf4d903c6287433d"
format_stamp: "Formatted at 2026-09-23 17:24:50 on dist-test-slave-ptl9"
server_key: "fd7dd9a9e5918a0b0962f64cf3cbfe4d"
server_key_iv: "cdb6827ef31079074314299b617ce589"
server_key_version: "encryptionkey@0"
I20260923 17:24:50.193988 14950 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:24:50.196331 14965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:50.197570 14950 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260923 17:24:50.197700 14950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-2/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-2/wal
uuid: "35ea8b120ee44e7bbf4d903c6287433d"
format_stamp: "Formatted at 2026-09-23 17:24:50 on dist-test-slave-ptl9"
server_key: "fd7dd9a9e5918a0b0962f64cf3cbfe4d"
server_key_iv: "cdb6827ef31079074314299b617ce589"
server_key_version: "encryptionkey@0"
I20260923 17:24:50.197809 14950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:50.210067 14950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:50.213434 14950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:50.213649 14950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:50.214236 14950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:24:50.215065 14950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:24:50.215135 14950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:50.215207 14950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:24:50.215250 14950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:50.223910 14950 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.3:44107
I20260923 17:24:50.223932 15078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.3:44107 every 8 connection(s)
I20260923 17:24:50.224830 14950 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
I20260923 17:24:50.231674 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 14950
I20260923 17:24:50.231766 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance
I20260923 17:24:50.232061 12180 external_mini_cluster.cc:1428] Setting key d757f383cfbba0212348dc66d9e1d467
Sep 23 17:24:50 dist-test-slave-ptl9 krb5kdc[14575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184290, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:24:50.238595 14659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:50.226637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:38885 (local address 127.11.229.62:46477)
0923 17:24:50.226809 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:50.226813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:50.227935 (+ 1122us) server_negotiation.cc:407] Connection header received
0923 17:24:50.228910 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:50.228913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:50.228960 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:50.229087 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:50.230222 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:50.230752 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:50.231917 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:50.232179 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:50.232305 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:50.235287 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:50.235290 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:50.235293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:50.235318 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:50.236571 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:50.237081 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:50.237084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:50.237085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:50.237163 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:50.237551 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:50.237553 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:50.237554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:50.237749 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:50.237828 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:50.238363 (+ 535us) server_negotiation.cc:299] Negotiation successful
0923 17:24:50.238471 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 17:24:50.239457 15081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:50.227025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46477 (local address 127.11.229.3:38885)
0923 17:24:50.227741 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:50.227790 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:50.228656 (+ 866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:50.229254 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:50.229275 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:50.229618 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:50.230084 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:50.230097 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:50.230859 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:50.230861 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:50.231775 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:50.231789 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:50.232344 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:50.232349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:50.232526 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:50.233320 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:50.233341 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:50.235121 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:50.236683 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:50.236688 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:50.236702 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:50.236988 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:50.237244 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:50.237246 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:50.237247 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:50.237455 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:50.237843 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:50.237849 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:50.238114 (+ 265us) client_negotiation.cc:769] Sending connection context
0923 17:24:50.238287 (+ 173us) client_negotiation.cc:240] Negotiation successful
0923 17:24:50.238574 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":113,"threads_started":1}
I20260923 17:24:50.240695 15079 heartbeater.cc:347] Connected to a master server at 127.11.229.62:46477
I20260923 17:24:50.240923 15079 heartbeater.cc:464] Registering TS with master...
I20260923 17:24:50.241470 15079 heartbeater.cc:511] Master 127.11.229.62:46477 requested a full tablet report, sending...
I20260923 17:24:50.242542 14620 ts_manager.cc:194] Registered new tserver with Master: 35ea8b120ee44e7bbf4d903c6287433d (127.11.229.3:44107)
I20260923 17:24:50.243093 14620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.3@KRBTEST.COM'} at 127.11.229.3:38885
I20260923 17:24:50.246237 12180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 17:24:50.259532 14659 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:24:50.258108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34920 (local address 127.11.229.62:46477)
0923 17:24:50.258253 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:50.258257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:50.258379 (+ 122us) server_negotiation.cc:407] Connection header received
0923 17:24:50.258500 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:50.258502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:50.258546 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:50.258625 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:50.259352 (+ 727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34920: BlockingRecv error: recv got EOF from 127.0.0.1:34920 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 17:24:50.260670 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 14682
I20260923 17:24:50.266531 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 14816
I20260923 17:24:50.272130 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 14950
I20260923 17:24:50.277890 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 14588
2026-09-23T17:24:50Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2154 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:24:50 dist-test-slave-ptl9 krb5kdc[15091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:24:50 dist-test-slave-ptl9 krb5kdc[15091](info): set up 2 sockets
Sep 23 17:24:50 dist-test-slave-ptl9 krb5kdc[15091](info): commencing operation
krb5kdc: starting...
W20260923 17:24:51.313522 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:24:51 dist-test-slave-ptl9 krb5kdc[15091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:24:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:24:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.229.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
I20260923 17:24:51.437397 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:45141
--webserver_interface=127.11.229.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:39791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:45141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:51.552646 15104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:51.552963 15104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:51.553073 15104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:51.557123 15104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:24:51.557220 15104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:51.557260 15104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:51.557308 15104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:24:51.557345 15104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:24:51.562517 15104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:39791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:45141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:45141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:51.563577 15104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:51.564594 15104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:51.573240 15110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:51.573299 15112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:51.573277 15109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:51.573544 15104 server_base.cc:1034] running on GCE node
I20260923 17:24:51.574146 15104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:51.575351 15104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:51.576530 15104 hybrid_clock.cc:648] HybridClock initialized: now 1790184291576509 us; error 32 us; skew 500 ppm
Sep 23 17:24:51 dist-test-slave-ptl9 krb5kdc[15091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184291, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:51.579510 15104 init.cc:377] Logged in from keytab as kudu/127.11.229.62@KRBTEST.COM (short username kudu)
I20260923 17:24:51.580519 15104 webserver.cc:460] Webserver started at http://127.11.229.62:42715/ using document root <none> and password file <none>
I20260923 17:24:51.581043 15104 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:51.581097 15104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:51.581308 15104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:51.583384 15104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "42dcfe237464467fae3bda62e0963461"
format_stamp: "Formatted at 2026-09-23 17:24:51 on dist-test-slave-ptl9"
server_key: "4017c3e53912a76439b5e997ef927ce3"
server_key_iv: "a947410e13b22eb5575f631fcc991033"
server_key_version: "encryptionkey@0"
I20260923 17:24:51.583880 15104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "42dcfe237464467fae3bda62e0963461"
format_stamp: "Formatted at 2026-09-23 17:24:51 on dist-test-slave-ptl9"
server_key: "4017c3e53912a76439b5e997ef927ce3"
server_key_iv: "a947410e13b22eb5575f631fcc991033"
server_key_version: "encryptionkey@0"
I20260923 17:24:51.587507 15104 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 17:24:51.589854 15119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:51.591135 15104 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:24:51.591269 15104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "42dcfe237464467fae3bda62e0963461"
format_stamp: "Formatted at 2026-09-23 17:24:51 on dist-test-slave-ptl9"
server_key: "4017c3e53912a76439b5e997ef927ce3"
server_key_iv: "a947410e13b22eb5575f631fcc991033"
server_key_version: "encryptionkey@0"
I20260923 17:24:51.591368 15104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:51.603725 15104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:51.606513 15104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:51.606724 15104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:51.613811 15104 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:45141
I20260923 17:24:51.614153 15171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:45141 every 8 connection(s)
I20260923 17:24:51.614796 15104 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:24:51.617777 15172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:51.623656 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 15104
I20260923 17:24:51.623755 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:24:51.623786 15172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42dcfe237464467fae3bda62e0963461: Bootstrap starting.
I20260923 17:24:51.624023 12180 external_mini_cluster.cc:1428] Setting key 6a3de9cf13388d4e139fc3bdc5b856c9
I20260923 17:24:51.626818 15172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42dcfe237464467fae3bda62e0963461: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:51.627772 15172 log.cc:824] T 00000000000000000000000000000000 P 42dcfe237464467fae3bda62e0963461: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:51.630072 15172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42dcfe237464467fae3bda62e0963461: No bootstrap required, opened a new log
Sep 23 17:24:51 dist-test-slave-ptl9 krb5kdc[15091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:24:51.634244 15172 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42dcfe237464467fae3bda62e0963461 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42dcfe237464467fae3bda62e0963461" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 45141 } }
I20260923 17:24:51.634524 15172 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42dcfe237464467fae3bda62e0963461 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:51.634595 15172 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42dcfe237464467fae3bda62e0963461 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42dcfe237464467fae3bda62e0963461, State: Initialized, Role: FOLLOWER
I20260923 17:24:51.635114 15172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42dcfe237464467fae3bda62e0963461 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42dcfe237464467fae3bda62e0963461" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 45141 } }
I20260923 17:24:51.635293 15172 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42dcfe237464467fae3bda62e0963461 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:24:51.635365 15172 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42dcfe237464467fae3bda62e0963461 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:24:51.635485 15172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42dcfe237464467fae3bda62e0963461 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:51.636377 15172 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42dcfe237464467fae3bda62e0963461 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42dcfe237464467fae3bda62e0963461" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 45141 } }
I20260923 17:24:51.636725 15172 leader_election.cc:302] T 00000000000000000000000000000000 P 42dcfe237464467fae3bda62e0963461 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42dcfe237464467fae3bda62e0963461; no voters:
I20260923 17:24:51.637090 15172 leader_election.cc:288] T 00000000000000000000000000000000 P 42dcfe237464467fae3bda62e0963461 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:24:51.637202 15177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42dcfe237464467fae3bda62e0963461 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:24:51.637450 15177 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42dcfe237464467fae3bda62e0963461 [term 1 LEADER]: Becoming Leader. State: Replica: 42dcfe237464467fae3bda62e0963461, State: Running, Role: LEADER
I20260923 17:24:51.637761 15177 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42dcfe237464467fae3bda62e0963461 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42dcfe237464467fae3bda62e0963461" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 45141 } }
I20260923 17:24:51.638489 15172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42dcfe237464467fae3bda62e0963461 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:24:51.639405 15178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42dcfe237464467fae3bda62e0963461 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42dcfe237464467fae3bda62e0963461" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42dcfe237464467fae3bda62e0963461" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 45141 } } }
I20260923 17:24:51.639580 15178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42dcfe237464467fae3bda62e0963461 [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:51.639916 15179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42dcfe237464467fae3bda62e0963461 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42dcfe237464467fae3bda62e0963461. Latest consensus state: current_term: 1 leader_uuid: "42dcfe237464467fae3bda62e0963461" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42dcfe237464467fae3bda62e0963461" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 45141 } } }
I20260923 17:24:51.640010 15179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42dcfe237464467fae3bda62e0963461 [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:51.640211 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:51.625401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50686 (local address 127.11.229.62:45141)
0923 17:24:51.625821 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:51.625844 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:51.625889 (+ 45us) server_negotiation.cc:407] Connection header received
0923 17:24:51.626834 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:51.626865 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:51.627350 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:51.627750 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:51.628651 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:51.630185 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:51.631055 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:51.631740 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:51.631900 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:51.634504 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:51.634510 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:51.634536 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:51.634575 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:51.636496 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:51.636977 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:51.636986 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:51.636994 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:51.637180 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:51.637548 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:51.637551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:51.637553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:51.638052 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:51.638262 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:51.638576 (+ 314us) server_negotiation.cc:299] Negotiation successful
0923 17:24:51.639059 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":91,"threads_started":1}
I20260923 17:24:51.643237 12180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260923 17:24:51.643831 15192 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 42dcfe237464467fae3bda62e0963461: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:24:51.643927 15192 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:24:51.643996 15186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:24:51.644654 15186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:24:51.650660 15186 catalog_manager.cc:1347] Generated new cluster ID: dd6e0f13464544bb811a1b080a32c6b6
I20260923 17:24:51.650730 15186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:24:51.665709 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:51.659871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50702 (local address 127.11.229.62:45141)
0923 17:24:51.659984 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:51.659988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:51.660256 (+ 268us) server_negotiation.cc:407] Connection header received
0923 17:24:51.660374 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:51.660378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:51.660434 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:51.660533 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:51.661100 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:51.661949 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:51.663097 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:51.663492 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:51.663596 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:51.664537 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:51.664541 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:51.664544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:24:51.665182 (+ 638us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:51.665220 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:24:51.665241 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:51.665369 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:51.665488 (+ 119us) server_negotiation.cc:299] Negotiation successful
0923 17:24:51.665552 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 17:24:51.671689 15186 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:24:51.672968 15186 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:24:51.683669 15186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42dcfe237464467fae3bda62e0963461: Generated new TSK 0
I20260923 17:24:51.684464 15186 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:24:51.695657 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 15104
2026-09-23T17:24:51Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1416 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:24:51 dist-test-slave-ptl9 krb5kdc[15205](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:24:51 dist-test-slave-ptl9 krb5kdc[15205](info): set up 2 sockets
Sep 23 17:24:51 dist-test-slave-ptl9 krb5kdc[15205](info): commencing operation
krb5kdc: starting...
W20260923 17:24:52.733541 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:24:52 dist-test-slave-ptl9 krb5kdc[15205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:24:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:24:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.229.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
I20260923 17:24:52.857148 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:37231
--webserver_interface=127.11.229.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:41845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:37231
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:52.968528 15218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:52.968835 15218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:52.968905 15218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:52.973582 15218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:24:52.973659 15218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:52.973695 15218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:52.973732 15218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:24:52.973773 15218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:24:52.979050 15218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:41845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:37231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:37231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:52.980158 15218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:52.981355 15218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:52.990172 15223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:52.990176 15226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:52.990199 15224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:52.990518 15218 server_base.cc:1034] running on GCE node
I20260923 17:24:52.991099 15218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:52.992177 15218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:52.993393 15218 hybrid_clock.cc:648] HybridClock initialized: now 1790184292993363 us; error 45 us; skew 500 ppm
Sep 23 17:24:52 dist-test-slave-ptl9 krb5kdc[15205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184292, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:52.996109 15218 init.cc:377] Logged in from keytab as kudu/127.11.229.62@KRBTEST.COM (short username kudu)
I20260923 17:24:52.997005 15218 webserver.cc:460] Webserver started at http://127.11.229.62:45989/ using document root <none> and password file <none>
I20260923 17:24:52.997630 15218 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:52.997695 15218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:52.997864 15218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:52.999945 15218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "4ff8e54fc18344c99f638cafcdbd483f"
format_stamp: "Formatted at 2026-09-23 17:24:52 on dist-test-slave-ptl9"
server_key: "b4e2a1d1a1271a10ba510bf8e3300c60"
server_key_iv: "fbbf4a0d3895ca3603e2d3340fe5e35a"
server_key_version: "encryptionkey@0"
I20260923 17:24:53.000478 15218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "4ff8e54fc18344c99f638cafcdbd483f"
format_stamp: "Formatted at 2026-09-23 17:24:52 on dist-test-slave-ptl9"
server_key: "b4e2a1d1a1271a10ba510bf8e3300c60"
server_key_iv: "fbbf4a0d3895ca3603e2d3340fe5e35a"
server_key_version: "encryptionkey@0"
I20260923 17:24:53.004441 15218 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260923 17:24:53.007027 15233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:53.008327 15218 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:24:53.008464 15218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "4ff8e54fc18344c99f638cafcdbd483f"
format_stamp: "Formatted at 2026-09-23 17:24:52 on dist-test-slave-ptl9"
server_key: "b4e2a1d1a1271a10ba510bf8e3300c60"
server_key_iv: "fbbf4a0d3895ca3603e2d3340fe5e35a"
server_key_version: "encryptionkey@0"
I20260923 17:24:53.008581 15218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:53.030222 15218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:53.033950 15218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:53.034173 15218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:53.041745 15218 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:37231
I20260923 17:24:53.042050 15285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:37231 every 8 connection(s)
I20260923 17:24:53.042804 15218 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:24:53.043771 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 15218
I20260923 17:24:53.043900 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:24:53.044170 12180 external_mini_cluster.cc:1428] Setting key 9ec88bfb8b0d303a907b21d2c91a264a
I20260923 17:24:53.046126 15286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:53.053158 15286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ff8e54fc18344c99f638cafcdbd483f: Bootstrap starting.
Sep 23 17:24:53 dist-test-slave-ptl9 krb5kdc[15205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:24:53.055816 15286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ff8e54fc18344c99f638cafcdbd483f: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:53.056593 15286 log.cc:824] T 00000000000000000000000000000000 P 4ff8e54fc18344c99f638cafcdbd483f: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:53.058625 15286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ff8e54fc18344c99f638cafcdbd483f: No bootstrap required, opened a new log
I20260923 17:24:53.059504 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:53.045681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52238 (local address 127.11.229.62:37231)
0923 17:24:53.046138 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:53.046157 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:53.046396 (+ 239us) server_negotiation.cc:407] Connection header received
0923 17:24:53.047190 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:53.047209 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:53.047544 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:53.047832 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:53.048783 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:53.049942 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:53.051193 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:53.051688 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:53.051812 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:53.054746 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:53.054752 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:53.054766 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:53.054796 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:53.056597 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:53.057075 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:53.057080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:53.057085 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:53.057194 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:53.057517 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:53.057523 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:53.057529 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:53.057920 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:53.058153 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:53.058432 (+ 279us) server_negotiation.cc:299] Negotiation successful
0923 17:24:53.058775 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":143,"threads_started":1}
I20260923 17:24:53.062227 15286 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ff8e54fc18344c99f638cafcdbd483f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ff8e54fc18344c99f638cafcdbd483f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 37231 } }
I20260923 17:24:53.062443 15286 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ff8e54fc18344c99f638cafcdbd483f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:53.062521 15286 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ff8e54fc18344c99f638cafcdbd483f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ff8e54fc18344c99f638cafcdbd483f, State: Initialized, Role: FOLLOWER
I20260923 17:24:53.062980 15286 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ff8e54fc18344c99f638cafcdbd483f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ff8e54fc18344c99f638cafcdbd483f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 37231 } }
I20260923 17:24:53.063131 15286 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ff8e54fc18344c99f638cafcdbd483f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:24:53.063195 15286 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ff8e54fc18344c99f638cafcdbd483f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:24:53.063285 15286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ff8e54fc18344c99f638cafcdbd483f [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:53.064328 15286 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ff8e54fc18344c99f638cafcdbd483f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ff8e54fc18344c99f638cafcdbd483f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 37231 } }
I20260923 17:24:53.064713 15286 leader_election.cc:302] T 00000000000000000000000000000000 P 4ff8e54fc18344c99f638cafcdbd483f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ff8e54fc18344c99f638cafcdbd483f; no voters:
I20260923 17:24:53.065083 15286 leader_election.cc:288] T 00000000000000000000000000000000 P 4ff8e54fc18344c99f638cafcdbd483f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:24:53.065243 15291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ff8e54fc18344c99f638cafcdbd483f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:24:53.065531 15291 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ff8e54fc18344c99f638cafcdbd483f [term 1 LEADER]: Becoming Leader. State: Replica: 4ff8e54fc18344c99f638cafcdbd483f, State: Running, Role: LEADER
I20260923 17:24:53.065863 15291 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ff8e54fc18344c99f638cafcdbd483f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ff8e54fc18344c99f638cafcdbd483f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 37231 } }
I20260923 17:24:53.066375 15286 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ff8e54fc18344c99f638cafcdbd483f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:24:53.067553 15293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ff8e54fc18344c99f638cafcdbd483f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ff8e54fc18344c99f638cafcdbd483f. Latest consensus state: current_term: 1 leader_uuid: "4ff8e54fc18344c99f638cafcdbd483f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ff8e54fc18344c99f638cafcdbd483f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 37231 } } }
I20260923 17:24:53.067685 15293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ff8e54fc18344c99f638cafcdbd483f [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:53.067992 15292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ff8e54fc18344c99f638cafcdbd483f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ff8e54fc18344c99f638cafcdbd483f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ff8e54fc18344c99f638cafcdbd483f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 37231 } } }
I20260923 17:24:53.068094 15292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ff8e54fc18344c99f638cafcdbd483f [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:53.068106 15300 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:24:53.071059 15300 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:24:53.076944 15300 catalog_manager.cc:1347] Generated new cluster ID: dde1532d782a47299588d37b4546cbcc
I20260923 17:24:53.077024 15300 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:24:53.088231 15300 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:24:53.089363 15300 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:24:53.096894 15300 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ff8e54fc18344c99f638cafcdbd483f: Generated new TSK 0
I20260923 17:24:53.097776 15300 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.229.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab.
Entry for principal kudu/127.11.229.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab.
I20260923 17:24:53.136485 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.1:0
--local_ip_for_outbound_sockets=127.11.229.1
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:37231
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:41845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:53.251564 15312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:53.251900 15312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:53.251976 15312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:53.256089 15312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:53.256153 15312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:53.256253 15312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.1
I20260923 17:24:53.261598 15312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:41845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:37231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.1
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:53.262804 15312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:53.263907 15312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:53.274274 15317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:53.274274 15318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:53.274322 15320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:53.274688 15312 server_base.cc:1034] running on GCE node
I20260923 17:24:53.275080 15312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:53.275749 15312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:53.276916 15312 hybrid_clock.cc:648] HybridClock initialized: now 1790184293276898 us; error 29 us; skew 500 ppm
Sep 23 17:24:53 dist-test-slave-ptl9 krb5kdc[15205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184293, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:53.285831 15312 init.cc:377] Logged in from keytab as kudu/127.11.229.1@KRBTEST.COM (short username kudu)
I20260923 17:24:53.286814 15312 webserver.cc:460] Webserver started at http://127.11.229.1:44545/ using document root <none> and password file <none>
I20260923 17:24:53.287458 15312 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:53.287555 15312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:53.287786 15312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:53.290055 15312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-0/data/instance:
uuid: "6034f0f34c154c169fc4cb3b1691e974"
format_stamp: "Formatted at 2026-09-23 17:24:53 on dist-test-slave-ptl9"
server_key: "e181cac7895e89c17d679e7e7bcb1cc2"
server_key_iv: "0b80c6727f1c830aadc1c244bc510458"
server_key_version: "encryptionkey@0"
I20260923 17:24:53.290709 15312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance:
uuid: "6034f0f34c154c169fc4cb3b1691e974"
format_stamp: "Formatted at 2026-09-23 17:24:53 on dist-test-slave-ptl9"
server_key: "e181cac7895e89c17d679e7e7bcb1cc2"
server_key_iv: "0b80c6727f1c830aadc1c244bc510458"
server_key_version: "encryptionkey@0"
I20260923 17:24:53.294608 15312 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:24:53.297140 15327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:53.298696 15312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:24:53.298866 15312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-0/wal
uuid: "6034f0f34c154c169fc4cb3b1691e974"
format_stamp: "Formatted at 2026-09-23 17:24:53 on dist-test-slave-ptl9"
server_key: "e181cac7895e89c17d679e7e7bcb1cc2"
server_key_iv: "0b80c6727f1c830aadc1c244bc510458"
server_key_version: "encryptionkey@0"
I20260923 17:24:53.299036 15312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:53.321218 15312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:53.324589 15312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:53.324832 15312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:53.325522 15312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:24:53.326424 15312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:24:53.326500 15312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:53.326591 15312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:24:53.326644 15312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:53.335546 15312 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.1:43165
I20260923 17:24:53.335572 15440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.1:43165 every 8 connection(s)
I20260923 17:24:53.336468 15312 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
I20260923 17:24:53.343521 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 15312
I20260923 17:24:53.343664 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance
I20260923 17:24:53.343953 12180 external_mini_cluster.cc:1428] Setting key cbabe0eda374a3eb574db45451e136e8
Sep 23 17:24:53 dist-test-slave-ptl9 krb5kdc[15205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184293, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:24:53.353029 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:53.338426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:54463 (local address 127.11.229.62:37231)
0923 17:24:53.338569 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:53.338573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:53.339612 (+ 1039us) server_negotiation.cc:407] Connection header received
0923 17:24:53.341620 (+ 2008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:53.341627 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:53.341761 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:53.341866 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:53.343189 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:53.344100 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:53.345602 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:53.345950 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:53.346078 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:53.348831 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:53.348838 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:53.348845 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:53.348881 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:53.350688 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:53.351210 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:53.351215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:53.351219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:53.351322 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:53.351721 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:53.351727 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:53.351730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:53.352014 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:53.352140 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:53.352648 (+ 508us) server_negotiation.cc:299] Negotiation successful
0923 17:24:53.352805 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 17:24:53.353820 15443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:53.338796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:37231 (local address 127.11.229.1:54463)
0923 17:24:53.339457 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:53.339502 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:53.341310 (+ 1808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:53.342062 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:53.342089 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:53.342522 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:53.343028 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:53.343042 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:53.344258 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:53.344261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:53.345500 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:53.345510 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:53.346125 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:53.346128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:53.346286 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:53.346918 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:53.346935 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:53.348660 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:53.350824 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:53.350829 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:53.350843 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:53.351098 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:53.351462 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:53.351465 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:53.351466 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:53.351612 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:53.352136 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:53.352141 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:53.352385 (+ 244us) client_negotiation.cc:769] Sending connection context
0923 17:24:53.352560 (+ 175us) client_negotiation.cc:240] Negotiation successful
0923 17:24:53.352828 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":174,"threads_started":1}
WARNING: no policy specified for kudu/127.11.229.2@KRBTEST.COM; defaulting to no policy
I20260923 17:24:53.355268 15441 heartbeater.cc:347] Connected to a master server at 127.11.229.62:37231
I20260923 17:24:53.355533 15441 heartbeater.cc:464] Registering TS with master...
I20260923 17:24:53.356096 15441 heartbeater.cc:511] Master 127.11.229.62:37231 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.2@KRBTEST.COM" created.
I20260923 17:24:53.358031 15250 ts_manager.cc:194] Registered new tserver with Master: 6034f0f34c154c169fc4cb3b1691e974 (127.11.229.1:43165)
I20260923 17:24:53.359191 15250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.1@KRBTEST.COM'} at 127.11.229.1:54463
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab.
Entry for principal kudu/127.11.229.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab.
I20260923 17:24:53.375900 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.2:0
--local_ip_for_outbound_sockets=127.11.229.2
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:37231
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:41845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:53.489584 15446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:53.489897 15446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:53.489976 15446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:53.494177 15446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:53.494259 15446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:53.494351 15446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.2
I20260923 17:24:53.499817 15446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:41845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:37231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.2
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:53.500967 15446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:53.502154 15446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:53.512375 15454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:53.512437 15452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:53.512403 15451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:53.512518 15446 server_base.cc:1034] running on GCE node
I20260923 17:24:53.513106 15446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:53.513792 15446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:53.514953 15446 hybrid_clock.cc:648] HybridClock initialized: now 1790184293514934 us; error 30 us; skew 500 ppm
Sep 23 17:24:53 dist-test-slave-ptl9 krb5kdc[15205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184293, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:53.517778 15446 init.cc:377] Logged in from keytab as kudu/127.11.229.2@KRBTEST.COM (short username kudu)
I20260923 17:24:53.518689 15446 webserver.cc:460] Webserver started at http://127.11.229.2:32873/ using document root <none> and password file <none>
I20260923 17:24:53.519271 15446 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:53.519359 15446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:53.519588 15446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:53.521754 15446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-1/data/instance:
uuid: "8e24f43b4edd4f7e8c1867fe9d51aaef"
format_stamp: "Formatted at 2026-09-23 17:24:53 on dist-test-slave-ptl9"
server_key: "08b4941744243d5e29625f7638c9dd51"
server_key_iv: "34980e8be8a3cdc2f9376d02afb19c16"
server_key_version: "encryptionkey@0"
I20260923 17:24:53.522320 15446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance:
uuid: "8e24f43b4edd4f7e8c1867fe9d51aaef"
format_stamp: "Formatted at 2026-09-23 17:24:53 on dist-test-slave-ptl9"
server_key: "08b4941744243d5e29625f7638c9dd51"
server_key_iv: "34980e8be8a3cdc2f9376d02afb19c16"
server_key_version: "encryptionkey@0"
I20260923 17:24:53.526014 15446 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:24:53.528396 15461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:53.529693 15446 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:24:53.529826 15446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-1/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-1/wal
uuid: "8e24f43b4edd4f7e8c1867fe9d51aaef"
format_stamp: "Formatted at 2026-09-23 17:24:53 on dist-test-slave-ptl9"
server_key: "08b4941744243d5e29625f7638c9dd51"
server_key_iv: "34980e8be8a3cdc2f9376d02afb19c16"
server_key_version: "encryptionkey@0"
I20260923 17:24:53.529937 15446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:53.565735 15446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:53.569167 15446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:53.569398 15446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:53.570025 15446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:24:53.570889 15446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:24:53.570959 15446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:53.571029 15446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:24:53.571079 15446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:53.580232 15446 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.2:37875
I20260923 17:24:53.580252 15574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.2:37875 every 8 connection(s)
I20260923 17:24:53.581175 15446 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
I20260923 17:24:53.582715 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 15446
I20260923 17:24:53.582799 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance
I20260923 17:24:53.583127 12180 external_mini_cluster.cc:1428] Setting key 229ebe3d6e0e17740348755c12e3f77b
Sep 23 17:24:53 dist-test-slave-ptl9 krb5kdc[15205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184293, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.229.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.3@KRBTEST.COM" created.
I20260923 17:24:53.596796 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:53.583245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:55939 (local address 127.11.229.62:37231)
0923 17:24:53.583386 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:53.583390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:53.584458 (+ 1068us) server_negotiation.cc:407] Connection header received
0923 17:24:53.585977 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:53.585980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:53.586034 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:53.586115 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:53.587394 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:53.587925 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:53.589810 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:53.590028 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:53.590110 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:53.593012 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:53.593023 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:53.593026 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:53.593075 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:53.594550 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:53.595048 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:53.595053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:53.595055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:53.595163 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:53.595569 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:53.595572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:53.595573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:53.595774 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:53.595880 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:53.596473 (+ 593us) server_negotiation.cc:299] Negotiation successful
0923 17:24:53.596614 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 17:24:53.597783 15577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:53.583740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:37231 (local address 127.11.229.2:55939)
0923 17:24:53.584301 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:53.584365 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:53.585769 (+ 1404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:53.586291 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:53.586298 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:53.586645 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:53.587219 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:53.587239 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:53.588061 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:53.588068 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:53.589678 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:53.589694 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:53.590202 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:53.590210 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:53.590463 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:53.591132 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:53.591164 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:53.592824 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:53.594674 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:53.594679 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:53.594695 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:53.594948 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:53.595320 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:53.595324 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:53.595325 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:53.595468 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:53.595880 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:53.595888 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:53.596204 (+ 316us) client_negotiation.cc:769] Sending connection context
0923 17:24:53.596422 (+ 218us) client_negotiation.cc:240] Negotiation successful
0923 17:24:53.596748 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":132,"threads_started":1}
I20260923 17:24:53.599084 15575 heartbeater.cc:347] Connected to a master server at 127.11.229.62:37231
I20260923 17:24:53.599350 15575 heartbeater.cc:464] Registering TS with master...
I20260923 17:24:53.599865 15575 heartbeater.cc:511] Master 127.11.229.62:37231 requested a full tablet report, sending...
I20260923 17:24:53.600987 15250 ts_manager.cc:194] Registered new tserver with Master: 8e24f43b4edd4f7e8c1867fe9d51aaef (127.11.229.2:37875)
I20260923 17:24:53.601626 15250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.2@KRBTEST.COM'} at 127.11.229.2:55939
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab.
Entry for principal kudu/127.11.229.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab.
I20260923 17:24:53.613188 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.3:0
--local_ip_for_outbound_sockets=127.11.229.3
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:37231
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:41845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:53.727120 15580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:53.727381 15580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:53.727424 15580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:53.731535 15580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:53.731594 15580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:53.731685 15580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.3
I20260923 17:24:53.737098 15580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:41845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:37231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.3
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:53.738231 15580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:53.739355 15580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:53.749126 15586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:53.749126 15588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:53.749131 15585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:53.749437 15580 server_base.cc:1034] running on GCE node
I20260923 17:24:53.749830 15580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:53.750473 15580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:53.751668 15580 hybrid_clock.cc:648] HybridClock initialized: now 1790184293751632 us; error 48 us; skew 500 ppm
Sep 23 17:24:53 dist-test-slave-ptl9 krb5kdc[15205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184293, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:53.754540 15580 init.cc:377] Logged in from keytab as kudu/127.11.229.3@KRBTEST.COM (short username kudu)
I20260923 17:24:53.755456 15580 webserver.cc:460] Webserver started at http://127.11.229.3:36491/ using document root <none> and password file <none>
I20260923 17:24:53.756027 15580 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:53.756137 15580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:53.756379 15580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:53.758509 15580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-2/data/instance:
uuid: "be2a47e24e824375a9e74d07c5bcd597"
format_stamp: "Formatted at 2026-09-23 17:24:53 on dist-test-slave-ptl9"
server_key: "239f18444436a0810f8b6b0d80b7c2b2"
server_key_iv: "1a645c328addd74b998973f55cade54d"
server_key_version: "encryptionkey@0"
I20260923 17:24:53.759124 15580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance:
uuid: "be2a47e24e824375a9e74d07c5bcd597"
format_stamp: "Formatted at 2026-09-23 17:24:53 on dist-test-slave-ptl9"
server_key: "239f18444436a0810f8b6b0d80b7c2b2"
server_key_iv: "1a645c328addd74b998973f55cade54d"
server_key_version: "encryptionkey@0"
I20260923 17:24:53.762976 15580 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:24:53.765578 15595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:53.766889 15580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:24:53.767031 15580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-2/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-2/wal
uuid: "be2a47e24e824375a9e74d07c5bcd597"
format_stamp: "Formatted at 2026-09-23 17:24:53 on dist-test-slave-ptl9"
server_key: "239f18444436a0810f8b6b0d80b7c2b2"
server_key_iv: "1a645c328addd74b998973f55cade54d"
server_key_version: "encryptionkey@0"
I20260923 17:24:53.767217 15580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:53.789126 15580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:53.792456 15580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:53.792665 15580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:53.793361 15580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:24:53.794214 15580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:24:53.794284 15580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:53.794353 15580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:24:53.794401 15580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:53.803742 15580 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.3:34619
I20260923 17:24:53.803763 15708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.3:34619 every 8 connection(s)
I20260923 17:24:53.804781 15580 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
I20260923 17:24:53.809963 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 15580
I20260923 17:24:53.810096 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance
I20260923 17:24:53.810423 12180 external_mini_cluster.cc:1428] Setting key 09b5326e6e1c8aab25a14127aa9de898
Sep 23 17:24:53 dist-test-slave-ptl9 krb5kdc[15205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184293, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:24:53.819941 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:53.806882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:44471 (local address 127.11.229.62:37231)
0923 17:24:53.807062 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:53.807066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:53.807927 (+ 861us) server_negotiation.cc:407] Connection header received
0923 17:24:53.809371 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:53.809374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:53.809424 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:53.809507 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:53.811172 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:53.811954 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:53.813581 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:53.813795 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:53.813878 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:53.816449 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:53.816452 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:53.816454 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:53.816480 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:53.817869 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:53.818376 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:53.818379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:53.818380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:53.818457 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:53.818816 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:53.818819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:53.818820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:53.819025 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:53.819114 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:53.819699 (+ 585us) server_negotiation.cc:299] Negotiation successful
0923 17:24:53.819815 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 17:24:53.820813 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:53.807277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:37231 (local address 127.11.229.3:44471)
0923 17:24:53.807756 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:53.807802 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:53.809064 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:53.809724 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:53.809735 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:53.810289 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:53.810991 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:53.811011 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:53.812099 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:53.812103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:53.813456 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:53.813474 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:53.813926 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:53.813929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:53.814083 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:53.814682 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:53.814698 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:53.816286 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:53.817983 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:53.817988 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:53.818002 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:53.818286 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:53.818537 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:53.818540 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:53.818541 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:53.818700 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:53.819154 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:53.819160 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:53.819411 (+ 251us) client_negotiation.cc:769] Sending connection context
0923 17:24:53.819598 (+ 187us) client_negotiation.cc:240] Negotiation successful
0923 17:24:53.819875 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":108,"threads_started":1}
I20260923 17:24:53.822141 15709 heartbeater.cc:347] Connected to a master server at 127.11.229.62:37231
I20260923 17:24:53.822410 15709 heartbeater.cc:464] Registering TS with master...
I20260923 17:24:53.822940 15709 heartbeater.cc:511] Master 127.11.229.62:37231 requested a full tablet report, sending...
I20260923 17:24:53.824133 15250 ts_manager.cc:194] Registered new tserver with Master: be2a47e24e824375a9e74d07c5bcd597 (127.11.229.3:34619)
I20260923 17:24:53.824993 15250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.3@KRBTEST.COM'} at 127.11.229.3:44471
I20260923 17:24:53.825469 12180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 17:24:53 dist-test-slave-ptl9 krb5kdc[15205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184293, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 23 17:24:53 dist-test-slave-ptl9 krb5kdc[15205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184293, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:24:53.851001 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:53.842676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52254 (local address 127.11.229.62:37231)
0923 17:24:53.842835 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:53.842841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:53.842991 (+ 150us) server_negotiation.cc:407] Connection header received
0923 17:24:53.843118 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:53.843121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:53.843174 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:53.843247 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:53.843918 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:53.844438 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:53.845245 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:53.845510 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:53.845590 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:53.848100 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:53.848103 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:53.848105 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:53.848131 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:53.849449 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:53.849830 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:53.849832 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:53.849834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:53.849910 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:53.850141 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:53.850155 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:53.850156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:53.850376 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:53.850481 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:53.850679 (+ 198us) server_negotiation.cc:299] Negotiation successful
0923 17:24:53.850779 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260923 17:24:53.851562 15250 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52254
I20260923 17:24:53.852403 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 15312
I20260923 17:24:53.858669 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 15446
I20260923 17:24:53.864434 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 15580
I20260923 17:24:53.870370 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 15218
2026-09-23T17:24:53Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2175 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:24:53 dist-test-slave-ptl9 krb5kdc[15721](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:24:53 dist-test-slave-ptl9 krb5kdc[15721](info): set up 2 sockets
Sep 23 17:24:53 dist-test-slave-ptl9 krb5kdc[15721](info): commencing operation
krb5kdc: starting...
W20260923 17:24:54.905634 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:24:54 dist-test-slave-ptl9 krb5kdc[15721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:24:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:24:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.229.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
I20260923 17:24:55.029676 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:39631
--webserver_interface=127.11.229.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:36857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:39631
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:55.153208 15734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:55.153540 15734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:55.153586 15734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:55.158007 15734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:24:55.158069 15734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:55.158088 15734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:55.158105 15734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:24:55.158121 15734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:24:55.163364 15734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:36857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:39631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:39631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:55.164353 15734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:55.165405 15734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:55.173782 15739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:55.174068 15734 server_base.cc:1034] running on GCE node
W20260923 17:24:55.173825 15742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:55.173799 15740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:55.174790 15734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:55.175935 15734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:55.177163 15734 hybrid_clock.cc:648] HybridClock initialized: now 1790184295177130 us; error 43 us; skew 500 ppm
Sep 23 17:24:55 dist-test-slave-ptl9 krb5kdc[15721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184295, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:55.180254 15734 init.cc:377] Logged in from keytab as kudu/127.11.229.62@KRBTEST.COM (short username kudu)
I20260923 17:24:55.181236 15734 webserver.cc:460] Webserver started at http://127.11.229.62:37859/ using document root <none> and password file <none>
I20260923 17:24:55.181872 15734 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:55.181963 15734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:55.182204 15734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:55.184300 15734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "1bd3d21d48584a4d96e9f2f02084c1d1"
format_stamp: "Formatted at 2026-09-23 17:24:55 on dist-test-slave-ptl9"
server_key: "4726ab41ac66b51a68918875c0408f6a"
server_key_iv: "613b557949deb1168d53d96768f870fe"
server_key_version: "encryptionkey@0"
I20260923 17:24:55.184871 15734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "1bd3d21d48584a4d96e9f2f02084c1d1"
format_stamp: "Formatted at 2026-09-23 17:24:55 on dist-test-slave-ptl9"
server_key: "4726ab41ac66b51a68918875c0408f6a"
server_key_iv: "613b557949deb1168d53d96768f870fe"
server_key_version: "encryptionkey@0"
I20260923 17:24:55.188874 15734 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 17:24:55.191390 15749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:55.192709 15734 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:24:55.192857 15734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "1bd3d21d48584a4d96e9f2f02084c1d1"
format_stamp: "Formatted at 2026-09-23 17:24:55 on dist-test-slave-ptl9"
server_key: "4726ab41ac66b51a68918875c0408f6a"
server_key_iv: "613b557949deb1168d53d96768f870fe"
server_key_version: "encryptionkey@0"
I20260923 17:24:55.192983 15734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:55.221427 15734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:55.224897 15734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:55.225131 15734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:55.233090 15734 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:39631
I20260923 17:24:55.233119 15801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:39631 every 8 connection(s)
I20260923 17:24:55.234187 15734 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:24:55.236848 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 15734
I20260923 17:24:55.236964 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:24:55.237288 12180 external_mini_cluster.cc:1428] Setting key 6d0c816b864c9f3042bba25fea6aa540
I20260923 17:24:55.237460 15802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:55.244081 15802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bd3d21d48584a4d96e9f2f02084c1d1: Bootstrap starting.
Sep 23 17:24:55 dist-test-slave-ptl9 krb5kdc[15721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:24:55.246886 15802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bd3d21d48584a4d96e9f2f02084c1d1: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:55.247960 15802 log.cc:824] T 00000000000000000000000000000000 P 1bd3d21d48584a4d96e9f2f02084c1d1: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:55.250370 15802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bd3d21d48584a4d96e9f2f02084c1d1: No bootstrap required, opened a new log
I20260923 17:24:55.252964 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:55.238794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54524 (local address 127.11.229.62:39631)
0923 17:24:55.239254 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:55.239269 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:55.239306 (+ 37us) server_negotiation.cc:407] Connection header received
0923 17:24:55.240261 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:55.240281 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:55.240612 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:55.241022 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:55.241845 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:55.242868 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:55.243758 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:55.244314 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:55.244477 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:55.247722 (+ 3245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:55.247728 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:55.247742 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:55.247771 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:55.249599 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:55.250114 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:55.250119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:55.250124 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:55.250236 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:55.250557 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:55.250561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:55.250563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:55.251088 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:55.251287 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:55.251564 (+ 277us) server_negotiation.cc:299] Negotiation successful
0923 17:24:55.251873 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":142,"threads_started":1}
I20260923 17:24:55.254951 15802 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bd3d21d48584a4d96e9f2f02084c1d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd3d21d48584a4d96e9f2f02084c1d1" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 39631 } }
I20260923 17:24:55.255234 15802 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bd3d21d48584a4d96e9f2f02084c1d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:55.255285 15802 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bd3d21d48584a4d96e9f2f02084c1d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bd3d21d48584a4d96e9f2f02084c1d1, State: Initialized, Role: FOLLOWER
I20260923 17:24:55.255767 15802 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bd3d21d48584a4d96e9f2f02084c1d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd3d21d48584a4d96e9f2f02084c1d1" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 39631 } }
I20260923 17:24:55.255891 15802 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bd3d21d48584a4d96e9f2f02084c1d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:24:55.255931 15802 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bd3d21d48584a4d96e9f2f02084c1d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:24:55.255998 15802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bd3d21d48584a4d96e9f2f02084c1d1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:55.256919 15802 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bd3d21d48584a4d96e9f2f02084c1d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd3d21d48584a4d96e9f2f02084c1d1" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 39631 } }
I20260923 17:24:55.257354 15802 leader_election.cc:302] T 00000000000000000000000000000000 P 1bd3d21d48584a4d96e9f2f02084c1d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bd3d21d48584a4d96e9f2f02084c1d1; no voters:
I20260923 17:24:55.257745 15802 leader_election.cc:288] T 00000000000000000000000000000000 P 1bd3d21d48584a4d96e9f2f02084c1d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:24:55.257874 15807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bd3d21d48584a4d96e9f2f02084c1d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:24:55.258145 15807 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bd3d21d48584a4d96e9f2f02084c1d1 [term 1 LEADER]: Becoming Leader. State: Replica: 1bd3d21d48584a4d96e9f2f02084c1d1, State: Running, Role: LEADER
I20260923 17:24:55.258538 15807 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bd3d21d48584a4d96e9f2f02084c1d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd3d21d48584a4d96e9f2f02084c1d1" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 39631 } }
I20260923 17:24:55.258997 15802 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bd3d21d48584a4d96e9f2f02084c1d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:24:55.260330 15809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bd3d21d48584a4d96e9f2f02084c1d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bd3d21d48584a4d96e9f2f02084c1d1. Latest consensus state: current_term: 1 leader_uuid: "1bd3d21d48584a4d96e9f2f02084c1d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd3d21d48584a4d96e9f2f02084c1d1" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 39631 } } }
I20260923 17:24:55.260504 15809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bd3d21d48584a4d96e9f2f02084c1d1 [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:55.260514 15808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bd3d21d48584a4d96e9f2f02084c1d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bd3d21d48584a4d96e9f2f02084c1d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd3d21d48584a4d96e9f2f02084c1d1" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 39631 } } }
I20260923 17:24:55.260967 15808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bd3d21d48584a4d96e9f2f02084c1d1 [sys.catalog]: This master's current role is: LEADER
W20260923 17:24:55.263609 15822 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1bd3d21d48584a4d96e9f2f02084c1d1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:24:55.263710 15822 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:24:55.263790 15819 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:24:55.264503 15819 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:24:55.270449 15819 catalog_manager.cc:1347] Generated new cluster ID: 92df9990cbfb45b79aac47d93095b7ac
I20260923 17:24:55.270515 15819 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:24:55.279882 15819 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:24:55.281217 15819 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:24:55.285965 15819 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bd3d21d48584a4d96e9f2f02084c1d1: Generated new TSK 0
I20260923 17:24:55.286620 15819 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.229.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab.
Entry for principal kudu/127.11.229.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab.
I20260923 17:24:55.320940 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.1:0
--local_ip_for_outbound_sockets=127.11.229.1
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:39631
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:36857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:55.444314 15828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:55.444595 15828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:55.444638 15828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:55.448808 15828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:55.448869 15828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:55.448971 15828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.1
I20260923 17:24:55.454351 15828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:36857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:39631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.1
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:55.455425 15828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:55.456478 15828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:55.466732 15836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:55.466846 15828 server_base.cc:1034] running on GCE node
W20260923 17:24:55.466876 15834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:55.467147 15833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:55.467581 15828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:55.468287 15828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:55.469475 15828 hybrid_clock.cc:648] HybridClock initialized: now 1790184295469442 us; error 47 us; skew 500 ppm
Sep 23 17:24:55 dist-test-slave-ptl9 krb5kdc[15721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184295, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:55.472198 15828 init.cc:377] Logged in from keytab as kudu/127.11.229.1@KRBTEST.COM (short username kudu)
I20260923 17:24:55.473099 15828 webserver.cc:460] Webserver started at http://127.11.229.1:41779/ using document root <none> and password file <none>
I20260923 17:24:55.473711 15828 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:55.473803 15828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:55.474042 15828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:55.476032 15828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-0/data/instance:
uuid: "6921aeca4ce44106ac551b4cd1b1c900"
format_stamp: "Formatted at 2026-09-23 17:24:55 on dist-test-slave-ptl9"
server_key: "e364acd2d6aa04813ef808890e92044e"
server_key_iv: "7e83de8746e2cc7418216ae683d71bba"
server_key_version: "encryptionkey@0"
I20260923 17:24:55.476608 15828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance:
uuid: "6921aeca4ce44106ac551b4cd1b1c900"
format_stamp: "Formatted at 2026-09-23 17:24:55 on dist-test-slave-ptl9"
server_key: "e364acd2d6aa04813ef808890e92044e"
server_key_iv: "7e83de8746e2cc7418216ae683d71bba"
server_key_version: "encryptionkey@0"
I20260923 17:24:55.480407 15828 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 17:24:55.482939 15843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:55.484122 15828 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:24:55.484259 15828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-0/wal
uuid: "6921aeca4ce44106ac551b4cd1b1c900"
format_stamp: "Formatted at 2026-09-23 17:24:55 on dist-test-slave-ptl9"
server_key: "e364acd2d6aa04813ef808890e92044e"
server_key_iv: "7e83de8746e2cc7418216ae683d71bba"
server_key_version: "encryptionkey@0"
I20260923 17:24:55.484378 15828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:55.504308 15828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:55.507356 15828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:55.507572 15828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:55.508275 15828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:24:55.509105 15828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:24:55.509176 15828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:55.509248 15828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:24:55.509321 15828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:55.518087 15828 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.1:35585
I20260923 17:24:55.518126 15956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.1:35585 every 8 connection(s)
I20260923 17:24:55.518936 15828 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
I20260923 17:24:55.528249 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 15828
Sep 23 17:24:55 dist-test-slave-ptl9 krb5kdc[15721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184295, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:24:55.528394 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance
I20260923 17:24:55.528700 12180 external_mini_cluster.cc:1428] Setting key c94e86f8fc802eab14d222a324b82e64
I20260923 17:24:55.532717 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:55.520715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:48283 (local address 127.11.229.62:39631)
0923 17:24:55.520886 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:55.520891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:55.521740 (+ 849us) server_negotiation.cc:407] Connection header received
0923 17:24:55.523073 (+ 1333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:55.523076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:55.523123 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:55.523198 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:55.524314 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:55.524881 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:55.526056 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:55.526287 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:55.526368 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:55.528966 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:55.528969 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:55.528972 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:55.528997 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:55.530633 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:55.531173 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:55.531178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:55.531179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:55.531265 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:55.531617 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:55.531622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:55.531624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:55.531811 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:55.531920 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:55.532424 (+ 504us) server_negotiation.cc:299] Negotiation successful
0923 17:24:55.532548 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 17:24:55.533844 15959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:55.521070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:39631 (local address 127.11.229.1:48283)
0923 17:24:55.521588 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:55.521630 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:55.522863 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:55.523361 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:55.523367 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:55.523719 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:55.524172 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:55.524184 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:55.525000 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:55.525003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:55.525949 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:55.525959 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:55.526416 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:55.526418 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:55.526561 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:55.527110 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:55.527126 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:55.528761 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:55.530758 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:55.530764 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:55.530779 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:55.531062 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:55.531367 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:55.531370 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:55.531372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:55.531516 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:55.531928 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:55.531936 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:55.532175 (+ 239us) client_negotiation.cc:769] Sending connection context
0923 17:24:55.532372 (+ 197us) client_negotiation.cc:240] Negotiation successful
0923 17:24:55.532691 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":133,"threads_started":1}
I20260923 17:24:55.535203 15957 heartbeater.cc:347] Connected to a master server at 127.11.229.62:39631
I20260923 17:24:55.535485 15957 heartbeater.cc:464] Registering TS with master...
I20260923 17:24:55.536149 15957 heartbeater.cc:511] Master 127.11.229.62:39631 requested a full tablet report, sending...
I20260923 17:24:55.537942 15766 ts_manager.cc:194] Registered new tserver with Master: 6921aeca4ce44106ac551b4cd1b1c900 (127.11.229.1:35585)
WARNING: no policy specified for kudu/127.11.229.2@KRBTEST.COM; defaulting to no policy
I20260923 17:24:55.539160 15766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.1@KRBTEST.COM'} at 127.11.229.1:48283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab.
Entry for principal kudu/127.11.229.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab.
I20260923 17:24:55.556779 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.2:0
--local_ip_for_outbound_sockets=127.11.229.2
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:39631
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:36857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:55.668097 15962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:55.668381 15962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:55.668457 15962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:55.672449 15962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:55.672538 15962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:55.672634 15962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.2
I20260923 17:24:55.677912 15962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:36857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:39631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.2
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:55.679021 15962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:55.680083 15962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:55.690336 15968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:55.690346 15970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:55.690346 15967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:55.690519 15962 server_base.cc:1034] running on GCE node
I20260923 17:24:55.691030 15962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:55.691674 15962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:55.692893 15962 hybrid_clock.cc:648] HybridClock initialized: now 1790184295692869 us; error 30 us; skew 500 ppm
Sep 23 17:24:55 dist-test-slave-ptl9 krb5kdc[15721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184295, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:55.695794 15962 init.cc:377] Logged in from keytab as kudu/127.11.229.2@KRBTEST.COM (short username kudu)
I20260923 17:24:55.696790 15962 webserver.cc:460] Webserver started at http://127.11.229.2:41345/ using document root <none> and password file <none>
I20260923 17:24:55.697412 15962 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:55.697475 15962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:55.697660 15962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:55.699764 15962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-1/data/instance:
uuid: "1d57f6dca1d24271836c6f3e89386cf9"
format_stamp: "Formatted at 2026-09-23 17:24:55 on dist-test-slave-ptl9"
server_key: "77a7cdaa495632218d6487d764b5a428"
server_key_iv: "6609875f36034c15b29a82f4ec3583eb"
server_key_version: "encryptionkey@0"
I20260923 17:24:55.700282 15962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance:
uuid: "1d57f6dca1d24271836c6f3e89386cf9"
format_stamp: "Formatted at 2026-09-23 17:24:55 on dist-test-slave-ptl9"
server_key: "77a7cdaa495632218d6487d764b5a428"
server_key_iv: "6609875f36034c15b29a82f4ec3583eb"
server_key_version: "encryptionkey@0"
I20260923 17:24:55.704193 15962 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 17:24:55.706727 15977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:55.707990 15962 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:24:55.708087 15962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-1/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-1/wal
uuid: "1d57f6dca1d24271836c6f3e89386cf9"
format_stamp: "Formatted at 2026-09-23 17:24:55 on dist-test-slave-ptl9"
server_key: "77a7cdaa495632218d6487d764b5a428"
server_key_iv: "6609875f36034c15b29a82f4ec3583eb"
server_key_version: "encryptionkey@0"
I20260923 17:24:55.708184 15962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:55.726454 15962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:55.729585 15962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:55.729749 15962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:55.730297 15962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:24:55.731081 15962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:24:55.731125 15962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:55.731165 15962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:24:55.731180 15962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:55.740501 15962 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.2:34887
I20260923 17:24:55.740581 16090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.2:34887 every 8 connection(s)
I20260923 17:24:55.741493 15962 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
I20260923 17:24:55.743206 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 15962
I20260923 17:24:55.743338 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance
I20260923 17:24:55.743652 12180 external_mini_cluster.cc:1428] Setting key 5d8de780637c180ba74eadfd4e9f8e02
Sep 23 17:24:55 dist-test-slave-ptl9 krb5kdc[15721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184295, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.229.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.3@KRBTEST.COM" created.
I20260923 17:24:55.756265 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:55.743463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:35697 (local address 127.11.229.62:39631)
0923 17:24:55.743613 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:55.743622 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:55.744661 (+ 1039us) server_negotiation.cc:407] Connection header received
0923 17:24:55.746146 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:55.746152 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:55.746214 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:55.746305 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:55.747444 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:55.747976 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:55.749223 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:55.749511 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:55.749626 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:55.752645 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:55.752649 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:55.752651 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:55.752676 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:55.754162 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:55.754684 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:55.754687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:55.754688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:55.754763 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:55.755059 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:55.755061 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:55.755062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:55.755294 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:55.755373 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:55.755924 (+ 551us) server_negotiation.cc:299] Negotiation successful
0923 17:24:55.756105 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 17:24:55.757195 16093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:55.743940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:39631 (local address 127.11.229.2:35697)
0923 17:24:55.744486 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:55.744543 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:55.745936 (+ 1393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:55.746460 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:55.746466 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:55.746824 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:55.747294 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:55.747307 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:55.748126 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:55.748130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:55.749102 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:55.749112 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:55.749665 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:55.749669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:55.749837 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:55.750626 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:55.750642 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:55.752455 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:55.754310 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:55.754315 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:55.754328 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:55.754581 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:55.754845 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:55.754847 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:55.754848 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:55.754980 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:55.755389 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:55.755394 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:55.755649 (+ 255us) client_negotiation.cc:769] Sending connection context
0923 17:24:55.755884 (+ 235us) client_negotiation.cc:240] Negotiation successful
0923 17:24:55.756259 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":143,"threads_started":1}
I20260923 17:24:55.758458 16091 heartbeater.cc:347] Connected to a master server at 127.11.229.62:39631
I20260923 17:24:55.758728 16091 heartbeater.cc:464] Registering TS with master...
I20260923 17:24:55.759323 16091 heartbeater.cc:511] Master 127.11.229.62:39631 requested a full tablet report, sending...
I20260923 17:24:55.760540 15766 ts_manager.cc:194] Registered new tserver with Master: 1d57f6dca1d24271836c6f3e89386cf9 (127.11.229.2:34887)
I20260923 17:24:55.761266 15766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.2@KRBTEST.COM'} at 127.11.229.2:35697
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab.
Entry for principal kudu/127.11.229.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab.
I20260923 17:24:55.772805 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.3:0
--local_ip_for_outbound_sockets=127.11.229.3
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:39631
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:36857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:55.884919 16096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:55.885190 16096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:55.885232 16096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:55.889417 16096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:55.889473 16096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:55.889567 16096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.3
I20260923 17:24:55.894942 16096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:36857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:39631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.3
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:55.896028 16096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:55.897150 16096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:55.907048 16104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:55.907058 16102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:55.907071 16101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:55.907112 16096 server_base.cc:1034] running on GCE node
I20260923 17:24:55.907737 16096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:55.908383 16096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:55.909571 16096 hybrid_clock.cc:648] HybridClock initialized: now 1790184295909555 us; error 29 us; skew 500 ppm
Sep 23 17:24:55 dist-test-slave-ptl9 krb5kdc[15721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184295, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:55.912348 16096 init.cc:377] Logged in from keytab as kudu/127.11.229.3@KRBTEST.COM (short username kudu)
I20260923 17:24:55.913319 16096 webserver.cc:460] Webserver started at http://127.11.229.3:35073/ using document root <none> and password file <none>
I20260923 17:24:55.913899 16096 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:55.913990 16096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:55.914209 16096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:55.916231 16096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-2/data/instance:
uuid: "78b63bfeb1a64c9fba5f05ae9363bc39"
format_stamp: "Formatted at 2026-09-23 17:24:55 on dist-test-slave-ptl9"
server_key: "7ed8de817d24b28accc4705e84e93d22"
server_key_iv: "d761371ebb44cac3078bd5c7c410a3dc"
server_key_version: "encryptionkey@0"
I20260923 17:24:55.916807 16096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance:
uuid: "78b63bfeb1a64c9fba5f05ae9363bc39"
format_stamp: "Formatted at 2026-09-23 17:24:55 on dist-test-slave-ptl9"
server_key: "7ed8de817d24b28accc4705e84e93d22"
server_key_iv: "d761371ebb44cac3078bd5c7c410a3dc"
server_key_version: "encryptionkey@0"
I20260923 17:24:55.920504 16096 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260923 17:24:55.922958 16111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:55.924204 16096 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:24:55.924351 16096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-2/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-2/wal
uuid: "78b63bfeb1a64c9fba5f05ae9363bc39"
format_stamp: "Formatted at 2026-09-23 17:24:55 on dist-test-slave-ptl9"
server_key: "7ed8de817d24b28accc4705e84e93d22"
server_key_iv: "d761371ebb44cac3078bd5c7c410a3dc"
server_key_version: "encryptionkey@0"
I20260923 17:24:55.924466 16096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:55.948462 16096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:55.951534 16096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:55.951747 16096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:55.952371 16096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:24:55.953249 16096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:24:55.953343 16096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:55.953408 16096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:24:55.953464 16096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:55.962877 16096 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.3:35381
I20260923 17:24:55.962903 16224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.3:35381 every 8 connection(s)
I20260923 17:24:55.963775 16096 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
I20260923 17:24:55.969223 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 16096
I20260923 17:24:55.969401 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance
I20260923 17:24:55.969758 12180 external_mini_cluster.cc:1428] Setting key 54f2f4ab570e98a0e6ee5a74aec31708
Sep 23 17:24:55 dist-test-slave-ptl9 krb5kdc[15721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184295, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:24:55.978591 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:55.965677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:57209 (local address 127.11.229.62:39631)
0923 17:24:55.965838 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:55.965842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:55.966706 (+ 864us) server_negotiation.cc:407] Connection header received
0923 17:24:55.968081 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:55.968085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:55.968142 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:55.968245 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:55.969493 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:55.970261 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:55.972169 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:55.972383 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:55.972466 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:55.975139 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:55.975142 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:55.975144 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:55.975168 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:55.976539 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:55.977041 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:55.977043 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:55.977045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:55.977118 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:55.977536 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:55.977539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:55.977540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:55.977723 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:55.977801 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:55.978360 (+ 559us) server_negotiation.cc:299] Negotiation successful
0923 17:24:55.978462 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 17:24:55.979399 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:55.966075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:39631 (local address 127.11.229.3:57209)
0923 17:24:55.966558 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:55.966599 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:55.967871 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:55.968417 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:55.968424 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:55.968764 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:55.969309 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:55.969333 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:55.970454 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:55.970458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:55.972034 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:55.972046 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:55.972526 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:55.972529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:55.972685 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:55.973418 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:55.973442 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:55.974969 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:55.976670 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:55.976675 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:55.976686 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:55.976942 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:55.977225 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:55.977228 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:55.977229 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:55.977436 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:55.977844 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:55.977850 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:55.978123 (+ 273us) client_negotiation.cc:769] Sending connection context
0923 17:24:55.978286 (+ 163us) client_negotiation.cc:240] Negotiation successful
0923 17:24:55.978552 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":109,"threads_started":1}
I20260923 17:24:55.980569 16225 heartbeater.cc:347] Connected to a master server at 127.11.229.62:39631
I20260923 17:24:55.980794 16225 heartbeater.cc:464] Registering TS with master...
I20260923 17:24:55.981348 16225 heartbeater.cc:511] Master 127.11.229.62:39631 requested a full tablet report, sending...
I20260923 17:24:55.982399 15766 ts_manager.cc:194] Registered new tserver with Master: 78b63bfeb1a64c9fba5f05ae9363bc39 (127.11.229.3:35381)
I20260923 17:24:55.983060 15766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.3@KRBTEST.COM'} at 127.11.229.3:57209
I20260923 17:24:55.983961 12180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 17:24:55 dist-test-slave-ptl9 krb5kdc[15721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 17:24:56 dist-test-slave-ptl9 krb5kdc[15721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.1@KRBTEST.COM
I20260923 17:24:56.010841 16232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:56.000970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32780 (local address 127.11.229.1:35585)
0923 17:24:56.001340 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:56.001359 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:56.001390 (+ 31us) server_negotiation.cc:407] Connection header received
0923 17:24:56.001468 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:56.001473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:56.001677 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:56.001842 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:56.002584 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:56.003366 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:56.004246 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:56.004637 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:56.004763 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:56.007298 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:56.007308 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:56.007321 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:56.007346 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:56.008914 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:56.009420 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:56.009426 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:56.009428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:56.009516 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:56.009803 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:56.009806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:56.009808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:56.010072 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:56.010315 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:56.010500 (+ 185us) server_negotiation.cc:299] Negotiation successful
0923 17:24:56.010662 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":109,"threads_started":1}
Sep 23 17:24:56 dist-test-slave-ptl9 krb5kdc[15721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:24:56.022224 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:56.014069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54540 (local address 127.11.229.62:39631)
0923 17:24:56.014265 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:56.014270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:56.014286 (+ 16us) server_negotiation.cc:407] Connection header received
0923 17:24:56.014351 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:56.014354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:56.014396 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:56.014469 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:56.015057 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:56.015600 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:56.016381 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:56.016606 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:56.016693 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:56.018959 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:56.018962 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:56.018964 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:56.018995 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:56.020416 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:56.020869 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:56.020872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:56.020873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:56.020962 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:56.021246 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:56.021249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:56.021250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:56.021543 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:56.021667 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:56.021891 (+ 224us) server_negotiation.cc:299] Negotiation successful
0923 17:24:56.022033 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260923 17:24:56.022665 15766 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54540
I20260923 17:24:56.023322 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 15828
I20260923 17:24:56.029783 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 15962
I20260923 17:24:56.035413 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 16096
I20260923 17:24:56.040548 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 15734
2026-09-23T17:24:56Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2169 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:24:56 dist-test-slave-ptl9 krb5kdc[16237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:24:56 dist-test-slave-ptl9 krb5kdc[16237](info): set up 2 sockets
Sep 23 17:24:56 dist-test-slave-ptl9 krb5kdc[16237](info): commencing operation
krb5kdc: starting...
W20260923 17:24:57.089582 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:24:57 dist-test-slave-ptl9 krb5kdc[16237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:24:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:24:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.229.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
I20260923 17:24:57.216427 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:42391
--webserver_interface=127.11.229.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:42391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:57.340847 16250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:57.341148 16250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:57.341228 16250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:57.345381 16250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:24:57.345476 16250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:57.345513 16250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:57.345580 16250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:24:57.345619 16250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:24:57.350912 16250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:42391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:42391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:57.352046 16250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:57.353137 16250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:57.362686 16256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:57.362707 16255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:57.362694 16258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:57.363015 16250 server_base.cc:1034] running on GCE node
I20260923 17:24:57.363657 16250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:57.364795 16250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:57.365986 16250 hybrid_clock.cc:648] HybridClock initialized: now 1790184297365973 us; error 36 us; skew 500 ppm
Sep 23 17:24:57 dist-test-slave-ptl9 krb5kdc[16237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184297, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:57.368736 16250 init.cc:377] Logged in from keytab as kudu/127.11.229.62@KRBTEST.COM (short username kudu)
I20260923 17:24:57.369606 16250 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:57.369699 16250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:57.369933 16250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:57.372004 16250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "93d6637c56454a1c80d723992f5fb834"
format_stamp: "Formatted at 2026-09-23 17:24:57 on dist-test-slave-ptl9"
server_key: "08b5ed23f4fb61861d64eb8d8c108a35"
server_key_iv: "ce52cef27f952e4fb6bfa00b9c7db4d6"
server_key_version: "encryptionkey@0"
I20260923 17:24:57.372591 16250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "93d6637c56454a1c80d723992f5fb834"
format_stamp: "Formatted at 2026-09-23 17:24:57 on dist-test-slave-ptl9"
server_key: "08b5ed23f4fb61861d64eb8d8c108a35"
server_key_iv: "ce52cef27f952e4fb6bfa00b9c7db4d6"
server_key_version: "encryptionkey@0"
I20260923 17:24:57.376459 16250 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260923 17:24:57.379025 16264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:57.380244 16250 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:24:57.380394 16250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "93d6637c56454a1c80d723992f5fb834"
format_stamp: "Formatted at 2026-09-23 17:24:57 on dist-test-slave-ptl9"
server_key: "08b5ed23f4fb61861d64eb8d8c108a35"
server_key_iv: "ce52cef27f952e4fb6bfa00b9c7db4d6"
server_key_version: "encryptionkey@0"
I20260923 17:24:57.380518 16250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:57.394901 16250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:57.398113 16250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:57.398327 16250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:57.405702 16250 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:42391
I20260923 17:24:57.405961 16316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:42391 every 8 connection(s)
I20260923 17:24:57.406431 16250 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:24:57.409188 16317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:57.412851 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 16250
I20260923 17:24:57.413024 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:24:57.413326 12180 external_mini_cluster.cc:1428] Setting key 229fc709ded14bac374ec1a7a63aa01f
I20260923 17:24:57.415406 16317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93d6637c56454a1c80d723992f5fb834: Bootstrap starting.
I20260923 17:24:57.417997 16317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93d6637c56454a1c80d723992f5fb834: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:57.418804 16317 log.cc:824] T 00000000000000000000000000000000 P 93d6637c56454a1c80d723992f5fb834: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:57.420994 16317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93d6637c56454a1c80d723992f5fb834: No bootstrap required, opened a new log
Sep 23 17:24:57 dist-test-slave-ptl9 krb5kdc[16237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:24:57.425046 16317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93d6637c56454a1c80d723992f5fb834 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d6637c56454a1c80d723992f5fb834" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 42391 } }
I20260923 17:24:57.425319 16317 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93d6637c56454a1c80d723992f5fb834 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:57.425386 16317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93d6637c56454a1c80d723992f5fb834 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93d6637c56454a1c80d723992f5fb834, State: Initialized, Role: FOLLOWER
I20260923 17:24:57.425899 16317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93d6637c56454a1c80d723992f5fb834 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d6637c56454a1c80d723992f5fb834" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 42391 } }
I20260923 17:24:57.426072 16317 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93d6637c56454a1c80d723992f5fb834 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:24:57.426159 16317 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93d6637c56454a1c80d723992f5fb834 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:24:57.426260 16317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93d6637c56454a1c80d723992f5fb834 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:57.427189 16317 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93d6637c56454a1c80d723992f5fb834 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d6637c56454a1c80d723992f5fb834" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 42391 } }
I20260923 17:24:57.427763 16317 leader_election.cc:302] T 00000000000000000000000000000000 P 93d6637c56454a1c80d723992f5fb834 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93d6637c56454a1c80d723992f5fb834; no voters:
I20260923 17:24:57.428228 16317 leader_election.cc:288] T 00000000000000000000000000000000 P 93d6637c56454a1c80d723992f5fb834 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:24:57.428298 16322 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93d6637c56454a1c80d723992f5fb834 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:24:57.428478 16320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:57.414784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57140 (local address 127.11.229.62:42391)
0923 17:24:57.415217 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:57.415245 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:57.415281 (+ 36us) server_negotiation.cc:407] Connection header received
0923 17:24:57.415972 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:57.415991 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:57.416335 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:57.416620 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:57.417543 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.418626 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.419690 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.420144 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.420258 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:57.422985 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:57.422990 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:57.423004 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:57.423030 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:57.424815 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:57.425327 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:57.425352 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:57.425356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:57.425577 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:57.425877 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:57.425881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:57.425882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:57.426428 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:57.426635 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:57.426941 (+ 306us) server_negotiation.cc:299] Negotiation successful
0923 17:24:57.427244 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":145,"threads_started":1}
I20260923 17:24:57.428505 16322 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93d6637c56454a1c80d723992f5fb834 [term 1 LEADER]: Becoming Leader. State: Replica: 93d6637c56454a1c80d723992f5fb834, State: Running, Role: LEADER
I20260923 17:24:57.429080 16322 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93d6637c56454a1c80d723992f5fb834 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d6637c56454a1c80d723992f5fb834" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 42391 } }
I20260923 17:24:57.429602 16317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93d6637c56454a1c80d723992f5fb834 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:24:57.430868 16323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93d6637c56454a1c80d723992f5fb834 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93d6637c56454a1c80d723992f5fb834" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d6637c56454a1c80d723992f5fb834" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 42391 } } }
I20260923 17:24:57.431000 16323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93d6637c56454a1c80d723992f5fb834 [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:57.431381 16324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93d6637c56454a1c80d723992f5fb834 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93d6637c56454a1c80d723992f5fb834. Latest consensus state: current_term: 1 leader_uuid: "93d6637c56454a1c80d723992f5fb834" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d6637c56454a1c80d723992f5fb834" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 42391 } } }
I20260923 17:24:57.432008 16324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93d6637c56454a1c80d723992f5fb834 [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:57.432264 16331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:24:57.435029 16331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:24:57.440793 16331 catalog_manager.cc:1347] Generated new cluster ID: 4e161d8c396548d4867fef213ee945f1
I20260923 17:24:57.440866 16331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:24:57.454901 16331 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:24:57.455889 16331 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:24:57.468807 16331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93d6637c56454a1c80d723992f5fb834: Generated new TSK 0
I20260923 17:24:57.469628 16331 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.229.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab.
Entry for principal kudu/127.11.229.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab.
I20260923 17:24:57.508265 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.1:0
--local_ip_for_outbound_sockets=127.11.229.1
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:42391
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:57.621243 16343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:57.621582 16343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:57.621628 16343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:57.625617 16343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:57.625669 16343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:57.625730 16343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.1
I20260923 17:24:57.631107 16343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:42391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.1
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:57.632198 16343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:57.633219 16343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:57.642699 16351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:57.642673 16349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:57.642689 16348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:57.643097 16343 server_base.cc:1034] running on GCE node
I20260923 17:24:57.643479 16343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:57.644091 16343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:57.645308 16343 hybrid_clock.cc:648] HybridClock initialized: now 1790184297645249 us; error 27 us; skew 500 ppm
Sep 23 17:24:57 dist-test-slave-ptl9 krb5kdc[16237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184297, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:57.648012 16343 init.cc:377] Logged in from keytab as kudu/127.11.229.1@KRBTEST.COM (short username kudu)
I20260923 17:24:57.648777 16343 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:57.648870 16343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:57.649109 16343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:57.651196 16343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-0/data/instance:
uuid: "c47cde25b26d48d5820a6459f7b03d2a"
format_stamp: "Formatted at 2026-09-23 17:24:57 on dist-test-slave-ptl9"
server_key: "e02e9360c436e544fcc060722676e604"
server_key_iv: "3d6bdea23fd9b4891232b4f427b44fca"
server_key_version: "encryptionkey@0"
I20260923 17:24:57.651770 16343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance:
uuid: "c47cde25b26d48d5820a6459f7b03d2a"
format_stamp: "Formatted at 2026-09-23 17:24:57 on dist-test-slave-ptl9"
server_key: "e02e9360c436e544fcc060722676e604"
server_key_iv: "3d6bdea23fd9b4891232b4f427b44fca"
server_key_version: "encryptionkey@0"
I20260923 17:24:57.655620 16343 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:24:57.658198 16357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:57.659458 16343 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:24:57.659600 16343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-0/wal
uuid: "c47cde25b26d48d5820a6459f7b03d2a"
format_stamp: "Formatted at 2026-09-23 17:24:57 on dist-test-slave-ptl9"
server_key: "e02e9360c436e544fcc060722676e604"
server_key_iv: "3d6bdea23fd9b4891232b4f427b44fca"
server_key_version: "encryptionkey@0"
I20260923 17:24:57.659718 16343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:57.691746 16343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:57.695228 16343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:57.695470 16343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:57.696153 16343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:24:57.697146 16343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:24:57.697222 16343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:57.697335 16343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:24:57.697383 16343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:57.707496 16343 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.1:39275
I20260923 17:24:57.707520 16470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.1:39275 every 8 connection(s)
I20260923 17:24:57.708277 16343 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
I20260923 17:24:57.715077 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 16343
I20260923 17:24:57.715224 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance
I20260923 17:24:57.715523 12180 external_mini_cluster.cc:1428] Setting key ca04b94aee1ccf6ed6ea4a580c5ccc2e
Sep 23 17:24:57 dist-test-slave-ptl9 krb5kdc[16237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184297, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:24:57.723408 16320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:57.710311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:60873 (local address 127.11.229.62:42391)
0923 17:24:57.710469 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:57.710473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:57.711460 (+ 987us) server_negotiation.cc:407] Connection header received
0923 17:24:57.712736 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:57.712739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:57.712791 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:57.712875 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:57.714120 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.714721 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.716127 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.716482 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.716575 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:57.719312 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:57.719316 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:57.719321 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:57.719352 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:57.720927 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:57.721615 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:57.721618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:57.721619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:57.721719 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:57.722143 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:57.722147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:57.722148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:57.722338 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:57.722418 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:57.723152 (+ 734us) server_negotiation.cc:299] Negotiation successful
0923 17:24:57.723267 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 17:24:57.724699 16473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:57.710798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:42391 (local address 127.11.229.1:60873)
0923 17:24:57.711297 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:57.711342 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:57.712526 (+ 1184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:57.713062 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:57.713069 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:57.713478 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:57.713972 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:57.713986 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.714872 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.714875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:57.715960 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:57.715972 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.716648 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.716652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:57.716865 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:57.717634 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:57.717652 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:57.719141 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:57.721060 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:57.721069 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:57.721090 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:57.721509 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:57.721826 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:57.721830 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:57.721832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:57.722034 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:57.722454 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:57.722464 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:57.722861 (+ 397us) client_negotiation.cc:769] Sending connection context
0923 17:24:57.723102 (+ 241us) client_negotiation.cc:240] Negotiation successful
0923 17:24:57.723498 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":130,"threads_started":1}
WARNING: no policy specified for kudu/127.11.229.2@KRBTEST.COM; defaulting to no policy
I20260923 17:24:57.726204 16471 heartbeater.cc:347] Connected to a master server at 127.11.229.62:42391
I20260923 17:24:57.726464 16471 heartbeater.cc:464] Registering TS with master...
I20260923 17:24:57.726996 16471 heartbeater.cc:511] Master 127.11.229.62:42391 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.2@KRBTEST.COM" created.
I20260923 17:24:57.728679 16281 ts_manager.cc:194] Registered new tserver with Master: c47cde25b26d48d5820a6459f7b03d2a (127.11.229.1:39275)
I20260923 17:24:57.729978 16281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.1@KRBTEST.COM'} at 127.11.229.1:60873
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab.
Entry for principal kudu/127.11.229.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab.
I20260923 17:24:57.744778 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.2:0
--local_ip_for_outbound_sockets=127.11.229.2
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:42391
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:57.855441 16476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:57.855681 16476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:57.855762 16476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:57.861814 16476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:57.861901 16476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:57.862023 16476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.2
I20260923 17:24:57.868448 16476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:42391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.2
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:57.869567 16476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:57.870570 16476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:57.880136 16481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:57.880158 16482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:57.880223 16476 server_base.cc:1034] running on GCE node
W20260923 17:24:57.880136 16484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:57.880759 16476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:57.881518 16476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:57.882730 16476 hybrid_clock.cc:648] HybridClock initialized: now 1790184297882712 us; error 32 us; skew 500 ppm
Sep 23 17:24:57 dist-test-slave-ptl9 krb5kdc[16237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184297, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:57.885653 16476 init.cc:377] Logged in from keytab as kudu/127.11.229.2@KRBTEST.COM (short username kudu)
I20260923 17:24:57.886380 16476 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:57.886469 16476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:57.886703 16476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:57.888753 16476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-1/data/instance:
uuid: "3dc3dafa5ce240fdbfa093cf031e9ebf"
format_stamp: "Formatted at 2026-09-23 17:24:57 on dist-test-slave-ptl9"
server_key: "1aadd1c41c4558fcc4f3b5cd8f1e7c09"
server_key_iv: "60d5db8212bc333643a86bfd92649975"
server_key_version: "encryptionkey@0"
I20260923 17:24:57.889389 16476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance:
uuid: "3dc3dafa5ce240fdbfa093cf031e9ebf"
format_stamp: "Formatted at 2026-09-23 17:24:57 on dist-test-slave-ptl9"
server_key: "1aadd1c41c4558fcc4f3b5cd8f1e7c09"
server_key_iv: "60d5db8212bc333643a86bfd92649975"
server_key_version: "encryptionkey@0"
I20260923 17:24:57.893234 16476 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:24:57.895816 16490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:57.897078 16476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:24:57.897217 16476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-1/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-1/wal
uuid: "3dc3dafa5ce240fdbfa093cf031e9ebf"
format_stamp: "Formatted at 2026-09-23 17:24:57 on dist-test-slave-ptl9"
server_key: "1aadd1c41c4558fcc4f3b5cd8f1e7c09"
server_key_iv: "60d5db8212bc333643a86bfd92649975"
server_key_version: "encryptionkey@0"
I20260923 17:24:57.897405 16476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:57.912200 16476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:57.915226 16476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:57.915431 16476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:57.916028 16476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:24:57.916895 16476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:24:57.916963 16476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:57.917033 16476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:24:57.917076 16476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:57.926064 16476 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.2:46215
I20260923 17:24:57.926085 16603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.2:46215 every 8 connection(s)
I20260923 17:24:57.926672 16476 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
I20260923 17:24:57.930900 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 16476
I20260923 17:24:57.931069 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance
I20260923 17:24:57.931375 12180 external_mini_cluster.cc:1428] Setting key 3087fbee366f72d6eed99fe7a5345623
Sep 23 17:24:57 dist-test-slave-ptl9 krb5kdc[16237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184297, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.229.3@KRBTEST.COM; defaulting to no policy
I20260923 17:24:57.942334 16320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:57.928384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:60943 (local address 127.11.229.62:42391)
0923 17:24:57.928544 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:57.928551 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:57.929391 (+ 840us) server_negotiation.cc:407] Connection header received
0923 17:24:57.931010 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:57.931014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:57.931079 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:57.931170 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:57.932726 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.933555 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.935005 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.935235 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.935316 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:57.938208 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:57.938214 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:57.938221 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:57.938259 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:57.939953 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:57.940527 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:57.940533 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:57.940537 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:57.940629 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:57.940968 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:57.940972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:57.940974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:57.941252 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:57.941401 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:57.942018 (+ 617us) server_negotiation.cc:299] Negotiation successful
0923 17:24:57.942154 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 17:24:57.943334 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:57.928795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:42391 (local address 127.11.229.2:60943)
0923 17:24:57.929237 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:57.929297 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:57.930757 (+ 1460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:57.931366 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:57.931374 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:57.931821 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:57.932545 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:57.932571 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.933691 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.933693 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:57.934872 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:57.934884 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:57.935402 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:57.935408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:57.935655 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:57.936296 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:57.936313 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:57.938016 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:57.940119 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:57.940125 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:57.940139 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:57.940415 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:57.940726 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:57.940728 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:57.940730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:57.940870 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:57.941416 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:57.941422 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:57.941694 (+ 272us) client_negotiation.cc:769] Sending connection context
0923 17:24:57.941948 (+ 254us) client_negotiation.cc:240] Negotiation successful
0923 17:24:57.942319 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.3@KRBTEST.COM" created.
I20260923 17:24:57.944926 16604 heartbeater.cc:347] Connected to a master server at 127.11.229.62:42391
I20260923 17:24:57.945201 16604 heartbeater.cc:464] Registering TS with master...
I20260923 17:24:57.945710 16604 heartbeater.cc:511] Master 127.11.229.62:42391 requested a full tablet report, sending...
I20260923 17:24:57.946806 16281 ts_manager.cc:194] Registered new tserver with Master: 3dc3dafa5ce240fdbfa093cf031e9ebf (127.11.229.2:46215)
I20260923 17:24:57.947499 16281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.2@KRBTEST.COM'} at 127.11.229.2:60943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab.
Entry for principal kudu/127.11.229.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab.
I20260923 17:24:57.960647 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.3:0
--local_ip_for_outbound_sockets=127.11.229.3
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:42391
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:58.074177 16609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:58.074443 16609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:58.074489 16609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:58.078471 16609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:58.078524 16609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:58.078585 16609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.3
I20260923 17:24:58.083689 16609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:42391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.3
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:58.084764 16609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:58.085834 16609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:58.095533 16617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:58.095583 16614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:58.095562 16615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:58.095808 16609 server_base.cc:1034] running on GCE node
I20260923 17:24:58.096362 16609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:58.097012 16609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:58.098203 16609 hybrid_clock.cc:648] HybridClock initialized: now 1790184298098146 us; error 79 us; skew 500 ppm
Sep 23 17:24:58 dist-test-slave-ptl9 krb5kdc[16237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184298, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:58.101086 16609 init.cc:377] Logged in from keytab as kudu/127.11.229.3@KRBTEST.COM (short username kudu)
I20260923 17:24:58.101991 16609 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:58.102094 16609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:58.102335 16609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:58.104456 16609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-2/data/instance:
uuid: "2b1f70ba4edc4fe4ba1e2fa7e54c9638"
format_stamp: "Formatted at 2026-09-23 17:24:58 on dist-test-slave-ptl9"
server_key: "4d238386396268baae4e431fdcfc5786"
server_key_iv: "d3279fea9b6816d4c0eff3726681c7cb"
server_key_version: "encryptionkey@0"
I20260923 17:24:58.105036 16609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance:
uuid: "2b1f70ba4edc4fe4ba1e2fa7e54c9638"
format_stamp: "Formatted at 2026-09-23 17:24:58 on dist-test-slave-ptl9"
server_key: "4d238386396268baae4e431fdcfc5786"
server_key_iv: "d3279fea9b6816d4c0eff3726681c7cb"
server_key_version: "encryptionkey@0"
I20260923 17:24:58.108865 16609 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260923 17:24:58.111341 16623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:58.112528 16609 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260923 17:24:58.112665 16609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-2/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-2/wal
uuid: "2b1f70ba4edc4fe4ba1e2fa7e54c9638"
format_stamp: "Formatted at 2026-09-23 17:24:58 on dist-test-slave-ptl9"
server_key: "4d238386396268baae4e431fdcfc5786"
server_key_iv: "d3279fea9b6816d4c0eff3726681c7cb"
server_key_version: "encryptionkey@0"
I20260923 17:24:58.112777 16609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:58.125627 16609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:58.128439 16609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:58.128650 16609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:58.129238 16609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:24:58.130228 16609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:24:58.130297 16609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:58.130367 16609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:24:58.130424 16609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:58.139516 16609 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.3:37715
I20260923 17:24:58.139539 16736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.3:37715 every 8 connection(s)
I20260923 17:24:58.140188 16609 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
I20260923 17:24:58.147166 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 16609
I20260923 17:24:58.147315 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance
I20260923 17:24:58.147600 12180 external_mini_cluster.cc:1428] Setting key 6709a9ac1348429084646935f6d67dac
Sep 23 17:24:58 dist-test-slave-ptl9 krb5kdc[16237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184298, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:24:58.153590 16320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:58.141796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:45785 (local address 127.11.229.62:42391)
0923 17:24:58.141941 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:58.141945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:58.142753 (+ 808us) server_negotiation.cc:407] Connection header received
0923 17:24:58.144059 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:58.144062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:58.144121 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:58.144229 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:58.145417 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:58.145947 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:58.147119 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:58.147426 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:58.147542 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:58.150361 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:58.150364 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:58.150367 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:58.150392 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:58.151740 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:58.152202 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:58.152205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:58.152206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:58.152280 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:58.152568 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:58.152571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:58.152572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:58.152746 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:58.152822 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:58.153364 (+ 542us) server_negotiation.cc:299] Negotiation successful
0923 17:24:58.153467 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 17:24:58.154387 16739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:58.142167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:42391 (local address 127.11.229.3:45785)
0923 17:24:58.142606 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:58.142648 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:58.143841 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:58.144404 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:58.144411 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:58.144763 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:58.145224 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:58.145237 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:58.146078 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:58.146082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:58.146992 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:58.147002 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:58.147586 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:58.147590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:58.147796 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:58.148458 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:58.148479 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:58.150212 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:58.151840 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:58.151845 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:58.151856 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:58.152099 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:58.152357 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:58.152359 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:58.152360 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:58.152484 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:58.152832 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:58.152837 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:58.153080 (+ 243us) client_negotiation.cc:769] Sending connection context
0923 17:24:58.153251 (+ 171us) client_negotiation.cc:240] Negotiation successful
0923 17:24:58.153565 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":105,"threads_started":1}
I20260923 17:24:58.155431 16737 heartbeater.cc:347] Connected to a master server at 127.11.229.62:42391
I20260923 17:24:58.155665 16737 heartbeater.cc:464] Registering TS with master...
I20260923 17:24:58.156105 16737 heartbeater.cc:511] Master 127.11.229.62:42391 requested a full tablet report, sending...
I20260923 17:24:58.157083 16281 ts_manager.cc:194] Registered new tserver with Master: 2b1f70ba4edc4fe4ba1e2fa7e54c9638 (127.11.229.3:37715)
I20260923 17:24:58.157699 16281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.3@KRBTEST.COM'} at 127.11.229.3:45785
I20260923 17:24:58.161342 12180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:24:58.170508 16320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:58.163255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57142 (local address 127.11.229.62:42391)
0923 17:24:58.163390 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:58.163395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:58.163574 (+ 179us) server_negotiation.cc:407] Connection header received
0923 17:24:58.163656 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:58.163658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:58.163694 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:58.163755 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:58.164471 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:58.164966 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:58.165823 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:58.166029 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:58.166107 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:58.167714 (+ 1607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:58.167720 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:58.167724 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:58.167756 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:58.169041 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:58.169480 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:58.169483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:58.169484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:58.169552 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:58.169756 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:58.169758 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:58.169759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:58.169946 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:58.170042 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:58.170219 (+ 177us) server_negotiation.cc:299] Negotiation successful
0923 17:24:58.170309 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 17:24:58.173167 16281 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:24:58.176239 16281 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:24:58.190340 16750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:58.184461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:39275 (local address 127.0.0.1:43892)
0923 17:24:58.185251 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:58.185297 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:58.185415 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:58.186157 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:58.186163 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:58.186198 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:24:58.186401 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:58.186408 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:58.187519 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:58.187525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:58.188779 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:58.188790 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:58.189809 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:58.189813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:58.189915 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:58.189950 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 17:24:58.190064 (+ 114us) client_negotiation.cc:240] Negotiation successful
0923 17:24:58.190181 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":60,"spinlock_wait_cycles":14336,"thread_start_us":95,"threads_started":1}
I20260923 17:24:58.190341 16749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:58.184632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43892 (local address 127.11.229.1:39275)
0923 17:24:58.185677 (+ 1045us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:58.185684 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:58.185702 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:24:58.185776 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:58.185781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:58.185982 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:58.186284 (+ 302us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:24:58.186504 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:58.187393 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:58.188918 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:58.189679 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:58.189797 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:58.189944 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:58.190094 (+ 150us) server_negotiation.cc:299] Negotiation successful
0923 17:24:58.190189 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":941,"thread_start_us":81,"threads_started":1}
I20260923 17:24:58.190800 16748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:58.184756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:37715 (local address 127.0.0.1:60986)
0923 17:24:58.185208 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:58.185286 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:58.185410 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:58.186157 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:58.186163 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:58.186196 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:24:58.186626 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:58.186633 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:58.188101 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:58.188104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:58.189300 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:58.189310 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:58.190396 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:58.190400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:58.190510 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:58.190544 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 17:24:58.190591 (+ 47us) client_negotiation.cc:240] Negotiation successful
0923 17:24:58.190653 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":72,"spinlock_wait_cycles":896,"thread_start_us":93,"threads_started":1}
I20260923 17:24:58.190858 16751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:58.185170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:46215 (local address 127.0.0.1:59012)
0923 17:24:58.185833 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:58.185848 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:58.185929 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:58.186649 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:58.186652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:58.186670 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:24:58.186758 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:58.186764 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:58.188128 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:58.188131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:58.189226 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:58.189235 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:58.190542 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:58.190548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:58.190630 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:58.190648 (+ 18us) client_negotiation.cc:769] Sending connection context
0923 17:24:58.190690 (+ 42us) client_negotiation.cc:240] Negotiation successful
0923 17:24:58.190744 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"mutex_wait_us":21,"thread_start_us":109,"threads_started":1}
I20260923 17:24:58.191454 16753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:58.185704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59012 (local address 127.11.229.2:46215)
0923 17:24:58.186190 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:58.186205 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:58.186227 (+ 22us) server_negotiation.cc:407] Connection header received
0923 17:24:58.186303 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:58.186308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:58.186505 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:58.186656 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:24:58.186876 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:58.187963 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:58.189430 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:58.190251 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:58.191007 (+ 756us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:58.191163 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:58.191214 (+ 51us) server_negotiation.cc:299] Negotiation successful
0923 17:24:58.191311 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":113,"threads_started":1}
I20260923 17:24:58.191884 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:58.185167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60986 (local address 127.11.229.3:37715)
0923 17:24:58.185665 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:58.185677 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:58.185701 (+ 24us) server_negotiation.cc:407] Connection header received
0923 17:24:58.185776 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:58.185781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:58.185982 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:58.186274 (+ 292us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:24:58.186753 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:58.187936 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:58.189429 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:58.190251 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:58.191125 (+ 874us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:58.191244 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:58.191284 (+ 40us) server_negotiation.cc:299] Negotiation successful
0923 17:24:58.191361 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":197,"threads_started":1}
I20260923 17:24:58.193703 16405 tablet_service.cc:1467] Processing CreateTablet for tablet 3ef84f5de382449a804bdc9ecb8c5169 (DEFAULT_TABLE table=test-table [id=170a3d16b2d44351a17a5631afd2c457]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:24:58.193822 16538 tablet_service.cc:1467] Processing CreateTablet for tablet 3ef84f5de382449a804bdc9ecb8c5169 (DEFAULT_TABLE table=test-table [id=170a3d16b2d44351a17a5631afd2c457]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:24:58.194788 16538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ef84f5de382449a804bdc9ecb8c5169. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:58.194788 16405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ef84f5de382449a804bdc9ecb8c5169. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:58.196102 16671 tablet_service.cc:1467] Processing CreateTablet for tablet 3ef84f5de382449a804bdc9ecb8c5169 (DEFAULT_TABLE table=test-table [id=170a3d16b2d44351a17a5631afd2c457]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:24:58.197042 16671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ef84f5de382449a804bdc9ecb8c5169. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:58.201767 16754 tablet_bootstrap.cc:492] T 3ef84f5de382449a804bdc9ecb8c5169 P c47cde25b26d48d5820a6459f7b03d2a: Bootstrap starting.
I20260923 17:24:58.203765 16756 tablet_bootstrap.cc:492] T 3ef84f5de382449a804bdc9ecb8c5169 P 2b1f70ba4edc4fe4ba1e2fa7e54c9638: Bootstrap starting.
I20260923 17:24:58.203928 16754 tablet_bootstrap.cc:654] T 3ef84f5de382449a804bdc9ecb8c5169 P c47cde25b26d48d5820a6459f7b03d2a: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:58.204564 16755 tablet_bootstrap.cc:492] T 3ef84f5de382449a804bdc9ecb8c5169 P 3dc3dafa5ce240fdbfa093cf031e9ebf: Bootstrap starting.
I20260923 17:24:58.205137 16754 log.cc:824] T 3ef84f5de382449a804bdc9ecb8c5169 P c47cde25b26d48d5820a6459f7b03d2a: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:58.205971 16756 tablet_bootstrap.cc:654] T 3ef84f5de382449a804bdc9ecb8c5169 P 2b1f70ba4edc4fe4ba1e2fa7e54c9638: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:58.206704 16755 tablet_bootstrap.cc:654] T 3ef84f5de382449a804bdc9ecb8c5169 P 3dc3dafa5ce240fdbfa093cf031e9ebf: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:58.207281 16754 tablet_bootstrap.cc:492] T 3ef84f5de382449a804bdc9ecb8c5169 P c47cde25b26d48d5820a6459f7b03d2a: No bootstrap required, opened a new log
I20260923 17:24:58.207341 16756 log.cc:824] T 3ef84f5de382449a804bdc9ecb8c5169 P 2b1f70ba4edc4fe4ba1e2fa7e54c9638: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:58.207494 16754 ts_tablet_manager.cc:1397] T 3ef84f5de382449a804bdc9ecb8c5169 P c47cde25b26d48d5820a6459f7b03d2a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 17:24:58.207865 16755 log.cc:824] T 3ef84f5de382449a804bdc9ecb8c5169 P 3dc3dafa5ce240fdbfa093cf031e9ebf: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:58.209249 16756 tablet_bootstrap.cc:492] T 3ef84f5de382449a804bdc9ecb8c5169 P 2b1f70ba4edc4fe4ba1e2fa7e54c9638: No bootstrap required, opened a new log
I20260923 17:24:58.209513 16756 ts_tablet_manager.cc:1397] T 3ef84f5de382449a804bdc9ecb8c5169 P 2b1f70ba4edc4fe4ba1e2fa7e54c9638: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260923 17:24:58.209829 16755 tablet_bootstrap.cc:492] T 3ef84f5de382449a804bdc9ecb8c5169 P 3dc3dafa5ce240fdbfa093cf031e9ebf: No bootstrap required, opened a new log
I20260923 17:24:58.210000 16755 ts_tablet_manager.cc:1397] T 3ef84f5de382449a804bdc9ecb8c5169 P 3dc3dafa5ce240fdbfa093cf031e9ebf: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260923 17:24:58.210985 16754 raft_consensus.cc:348] T 3ef84f5de382449a804bdc9ecb8c5169 P c47cde25b26d48d5820a6459f7b03d2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47cde25b26d48d5820a6459f7b03d2a" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 39275 } } peers { permanent_uuid: "2b1f70ba4edc4fe4ba1e2fa7e54c9638" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 37715 } } peers { permanent_uuid: "3dc3dafa5ce240fdbfa093cf031e9ebf" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 46215 } }
I20260923 17:24:58.211234 16754 raft_consensus.cc:374] T 3ef84f5de382449a804bdc9ecb8c5169 P c47cde25b26d48d5820a6459f7b03d2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:58.211297 16754 raft_consensus.cc:729] T 3ef84f5de382449a804bdc9ecb8c5169 P c47cde25b26d48d5820a6459f7b03d2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c47cde25b26d48d5820a6459f7b03d2a, State: Initialized, Role: FOLLOWER
I20260923 17:24:58.211747 16754 consensus_queue.cc:260] T 3ef84f5de382449a804bdc9ecb8c5169 P c47cde25b26d48d5820a6459f7b03d2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47cde25b26d48d5820a6459f7b03d2a" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 39275 } } peers { permanent_uuid: "2b1f70ba4edc4fe4ba1e2fa7e54c9638" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 37715 } } peers { permanent_uuid: "3dc3dafa5ce240fdbfa093cf031e9ebf" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 46215 } }
I20260923 17:24:58.212383 16471 heartbeater.cc:503] Master 127.11.229.62:42391 was elected leader, sending a full tablet report...
I20260923 17:24:58.212738 16754 ts_tablet_manager.cc:1428] T 3ef84f5de382449a804bdc9ecb8c5169 P c47cde25b26d48d5820a6459f7b03d2a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260923 17:24:58.213353 16756 raft_consensus.cc:348] T 3ef84f5de382449a804bdc9ecb8c5169 P 2b1f70ba4edc4fe4ba1e2fa7e54c9638 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47cde25b26d48d5820a6459f7b03d2a" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 39275 } } peers { permanent_uuid: "2b1f70ba4edc4fe4ba1e2fa7e54c9638" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 37715 } } peers { permanent_uuid: "3dc3dafa5ce240fdbfa093cf031e9ebf" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 46215 } }
I20260923 17:24:58.213625 16756 raft_consensus.cc:374] T 3ef84f5de382449a804bdc9ecb8c5169 P 2b1f70ba4edc4fe4ba1e2fa7e54c9638 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:58.213683 16756 raft_consensus.cc:729] T 3ef84f5de382449a804bdc9ecb8c5169 P 2b1f70ba4edc4fe4ba1e2fa7e54c9638 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b1f70ba4edc4fe4ba1e2fa7e54c9638, State: Initialized, Role: FOLLOWER
I20260923 17:24:58.213948 16755 raft_consensus.cc:348] T 3ef84f5de382449a804bdc9ecb8c5169 P 3dc3dafa5ce240fdbfa093cf031e9ebf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47cde25b26d48d5820a6459f7b03d2a" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 39275 } } peers { permanent_uuid: "2b1f70ba4edc4fe4ba1e2fa7e54c9638" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 37715 } } peers { permanent_uuid: "3dc3dafa5ce240fdbfa093cf031e9ebf" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 46215 } }
I20260923 17:24:58.214191 16755 raft_consensus.cc:374] T 3ef84f5de382449a804bdc9ecb8c5169 P 3dc3dafa5ce240fdbfa093cf031e9ebf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:58.214236 16755 raft_consensus.cc:729] T 3ef84f5de382449a804bdc9ecb8c5169 P 3dc3dafa5ce240fdbfa093cf031e9ebf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dc3dafa5ce240fdbfa093cf031e9ebf, State: Initialized, Role: FOLLOWER
I20260923 17:24:58.214643 16755 consensus_queue.cc:260] T 3ef84f5de382449a804bdc9ecb8c5169 P 3dc3dafa5ce240fdbfa093cf031e9ebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47cde25b26d48d5820a6459f7b03d2a" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 39275 } } peers { permanent_uuid: "2b1f70ba4edc4fe4ba1e2fa7e54c9638" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 37715 } } peers { permanent_uuid: "3dc3dafa5ce240fdbfa093cf031e9ebf" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 46215 } }
I20260923 17:24:58.215301 16604 heartbeater.cc:503] Master 127.11.229.62:42391 was elected leader, sending a full tablet report...
I20260923 17:24:58.215660 16755 ts_tablet_manager.cc:1428] T 3ef84f5de382449a804bdc9ecb8c5169 P 3dc3dafa5ce240fdbfa093cf031e9ebf: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260923 17:24:58.215955 16756 consensus_queue.cc:260] T 3ef84f5de382449a804bdc9ecb8c5169 P 2b1f70ba4edc4fe4ba1e2fa7e54c9638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47cde25b26d48d5820a6459f7b03d2a" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 39275 } } peers { permanent_uuid: "2b1f70ba4edc4fe4ba1e2fa7e54c9638" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 37715 } } peers { permanent_uuid: "3dc3dafa5ce240fdbfa093cf031e9ebf" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 46215 } }
I20260923 17:24:58.216565 16737 heartbeater.cc:503] Master 127.11.229.62:42391 was elected leader, sending a full tablet report...
I20260923 17:24:58.217598 16756 ts_tablet_manager.cc:1428] T 3ef84f5de382449a804bdc9ecb8c5169 P 2b1f70ba4edc4fe4ba1e2fa7e54c9638: Time spent starting tablet: real 0.008s user 0.003s sys 0.002s
I20260923 17:24:58.248265 16762 raft_consensus.cc:482] T 3ef84f5de382449a804bdc9ecb8c5169 P 2b1f70ba4edc4fe4ba1e2fa7e54c9638 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:24:58.248478 16762 raft_consensus.cc:504] T 3ef84f5de382449a804bdc9ecb8c5169 P 2b1f70ba4edc4fe4ba1e2fa7e54c9638 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47cde25b26d48d5820a6459f7b03d2a" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 39275 } } peers { permanent_uuid: "2b1f70ba4edc4fe4ba1e2fa7e54c9638" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 37715 } } peers { permanent_uuid: "3dc3dafa5ce240fdbfa093cf031e9ebf" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 46215 } }
I20260923 17:24:58.249679 16762 leader_election.cc:288] T 3ef84f5de382449a804bdc9ecb8c5169 P 2b1f70ba4edc4fe4ba1e2fa7e54c9638 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c47cde25b26d48d5820a6459f7b03d2a (127.11.229.1:39275), 3dc3dafa5ce240fdbfa093cf031e9ebf (127.11.229.2:46215)
I20260923 17:24:58.254715 16753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:58.250069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:46775 (local address 127.11.229.2:46215)
0923 17:24:58.250212 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:58.250218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:58.250417 (+ 199us) server_negotiation.cc:407] Connection header received
0923 17:24:58.250475 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:58.250480 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:58.250532 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:58.250601 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:24:58.250853 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:58.251633 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:58.253473 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:58.254087 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:58.254211 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:58.254263 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:58.254509 (+ 246us) server_negotiation.cc:299] Negotiation successful
0923 17:24:58.254581 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 17:24:58.254715 16749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:58.249976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:51237 (local address 127.11.229.1:39275)
0923 17:24:58.250132 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:58.250137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:58.250226 (+ 89us) server_negotiation.cc:407] Connection header received
0923 17:24:58.250360 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:58.250363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:58.250436 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:58.250528 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:24:58.250850 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:58.251636 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:58.253473 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:58.254087 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:58.254211 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:58.254263 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:58.254519 (+ 256us) server_negotiation.cc:299] Negotiation successful
0923 17:24:58.254584 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 17:24:58.254763 16739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:58.249889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:39275 (local address 127.11.229.3:51237)
0923 17:24:58.250131 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:58.250149 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:58.250254 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:58.250543 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:58.250545 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:58.250559 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:24:58.250714 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:58.250723 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:58.251744 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:58.251747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:58.253347 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:58.253359 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:58.254209 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:58.254212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:58.254388 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:58.254402 (+ 14us) client_negotiation.cc:769] Sending connection context
0923 17:24:58.254479 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 17:24:58.254593 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":211}
I20260923 17:24:58.254776 16763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:58.249915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:46215 (local address 127.11.229.3:46775)
0923 17:24:58.250312 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:58.250326 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:58.250411 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:58.250634 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:58.250637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:58.250648 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:24:58.250747 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:58.250756 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:58.251744 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:58.251747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:58.253347 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:58.253359 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:58.254209 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:58.254212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:58.254361 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:58.254402 (+ 41us) client_negotiation.cc:769] Sending connection context
0923 17:24:58.254479 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 17:24:58.254593 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":243,"thread_start_us":123,"threads_started":1}
I20260923 17:24:58.255417 16425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ef84f5de382449a804bdc9ecb8c5169" candidate_uuid: "2b1f70ba4edc4fe4ba1e2fa7e54c9638" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c47cde25b26d48d5820a6459f7b03d2a" is_pre_election: true
I20260923 17:24:58.255530 16558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ef84f5de382449a804bdc9ecb8c5169" candidate_uuid: "2b1f70ba4edc4fe4ba1e2fa7e54c9638" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dc3dafa5ce240fdbfa093cf031e9ebf" is_pre_election: true
I20260923 17:24:58.255753 16425 raft_consensus.cc:2454] T 3ef84f5de382449a804bdc9ecb8c5169 P c47cde25b26d48d5820a6459f7b03d2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b1f70ba4edc4fe4ba1e2fa7e54c9638 in term 0.
I20260923 17:24:58.255865 16558 raft_consensus.cc:2454] T 3ef84f5de382449a804bdc9ecb8c5169 P 3dc3dafa5ce240fdbfa093cf031e9ebf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b1f70ba4edc4fe4ba1e2fa7e54c9638 in term 0.
I20260923 17:24:58.256229 16624 leader_election.cc:302] T 3ef84f5de382449a804bdc9ecb8c5169 P 2b1f70ba4edc4fe4ba1e2fa7e54c9638 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b1f70ba4edc4fe4ba1e2fa7e54c9638, c47cde25b26d48d5820a6459f7b03d2a; no voters:
I20260923 17:24:58.256454 16762 raft_consensus.cc:2781] T 3ef84f5de382449a804bdc9ecb8c5169 P 2b1f70ba4edc4fe4ba1e2fa7e54c9638 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:24:58.256537 16762 raft_consensus.cc:482] T 3ef84f5de382449a804bdc9ecb8c5169 P 2b1f70ba4edc4fe4ba1e2fa7e54c9638 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:24:58.256580 16762 raft_consensus.cc:3037] T 3ef84f5de382449a804bdc9ecb8c5169 P 2b1f70ba4edc4fe4ba1e2fa7e54c9638 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:58.257643 16762 raft_consensus.cc:504] T 3ef84f5de382449a804bdc9ecb8c5169 P 2b1f70ba4edc4fe4ba1e2fa7e54c9638 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47cde25b26d48d5820a6459f7b03d2a" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 39275 } } peers { permanent_uuid: "2b1f70ba4edc4fe4ba1e2fa7e54c9638" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 37715 } } peers { permanent_uuid: "3dc3dafa5ce240fdbfa093cf031e9ebf" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 46215 } }
I20260923 17:24:58.257946 16762 leader_election.cc:288] T 3ef84f5de382449a804bdc9ecb8c5169 P 2b1f70ba4edc4fe4ba1e2fa7e54c9638 [CANDIDATE]: Term 1 election: Requested vote from peers c47cde25b26d48d5820a6459f7b03d2a (127.11.229.1:39275), 3dc3dafa5ce240fdbfa093cf031e9ebf (127.11.229.2:46215)
I20260923 17:24:58.258292 16425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ef84f5de382449a804bdc9ecb8c5169" candidate_uuid: "2b1f70ba4edc4fe4ba1e2fa7e54c9638" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c47cde25b26d48d5820a6459f7b03d2a"
I20260923 17:24:58.258314 16558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ef84f5de382449a804bdc9ecb8c5169" candidate_uuid: "2b1f70ba4edc4fe4ba1e2fa7e54c9638" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dc3dafa5ce240fdbfa093cf031e9ebf"
I20260923 17:24:58.258394 16425 raft_consensus.cc:3037] T 3ef84f5de382449a804bdc9ecb8c5169 P c47cde25b26d48d5820a6459f7b03d2a [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:58.258436 16558 raft_consensus.cc:3037] T 3ef84f5de382449a804bdc9ecb8c5169 P 3dc3dafa5ce240fdbfa093cf031e9ebf [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:58.259358 16425 raft_consensus.cc:2454] T 3ef84f5de382449a804bdc9ecb8c5169 P c47cde25b26d48d5820a6459f7b03d2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b1f70ba4edc4fe4ba1e2fa7e54c9638 in term 1.
I20260923 17:24:58.259507 16558 raft_consensus.cc:2454] T 3ef84f5de382449a804bdc9ecb8c5169 P 3dc3dafa5ce240fdbfa093cf031e9ebf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b1f70ba4edc4fe4ba1e2fa7e54c9638 in term 1.
I20260923 17:24:58.259676 16624 leader_election.cc:302] T 3ef84f5de382449a804bdc9ecb8c5169 P 2b1f70ba4edc4fe4ba1e2fa7e54c9638 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b1f70ba4edc4fe4ba1e2fa7e54c9638, c47cde25b26d48d5820a6459f7b03d2a; no voters:
I20260923 17:24:58.259856 16762 raft_consensus.cc:2781] T 3ef84f5de382449a804bdc9ecb8c5169 P 2b1f70ba4edc4fe4ba1e2fa7e54c9638 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:24:58.260092 16762 raft_consensus.cc:686] T 3ef84f5de382449a804bdc9ecb8c5169 P 2b1f70ba4edc4fe4ba1e2fa7e54c9638 [term 1 LEADER]: Becoming Leader. State: Replica: 2b1f70ba4edc4fe4ba1e2fa7e54c9638, State: Running, Role: LEADER
I20260923 17:24:58.260439 16762 consensus_queue.cc:237] T 3ef84f5de382449a804bdc9ecb8c5169 P 2b1f70ba4edc4fe4ba1e2fa7e54c9638 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47cde25b26d48d5820a6459f7b03d2a" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 39275 } } peers { permanent_uuid: "2b1f70ba4edc4fe4ba1e2fa7e54c9638" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 37715 } } peers { permanent_uuid: "3dc3dafa5ce240fdbfa093cf031e9ebf" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 46215 } }
I20260923 17:24:58.263906 16280 catalog_manager.cc:5495] T 3ef84f5de382449a804bdc9ecb8c5169 P 2b1f70ba4edc4fe4ba1e2fa7e54c9638 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b1f70ba4edc4fe4ba1e2fa7e54c9638 (127.11.229.3). New cstate: current_term: 1 leader_uuid: "2b1f70ba4edc4fe4ba1e2fa7e54c9638" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47cde25b26d48d5820a6459f7b03d2a" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 39275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b1f70ba4edc4fe4ba1e2fa7e54c9638" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 37715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3dc3dafa5ce240fdbfa093cf031e9ebf" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 46215 } health_report { overall_health: UNKNOWN } } }
I20260923 17:24:58.288144 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:58.284769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60988 (local address 127.11.229.3:37715)
0923 17:24:58.284914 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:58.284918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:58.284932 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:24:58.285129 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:58.285132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:58.285191 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:58.285334 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:24:58.285547 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:58.286039 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:58.286849 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:58.287054 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:58.287136 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:58.287393 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:24:58.287807 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:24:58.287900 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:58.287955 (+ 55us) server_negotiation.cc:299] Negotiation successful
0923 17:24:58.288006 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 17:24:58.294123 16425 raft_consensus.cc:1261] T 3ef84f5de382449a804bdc9ecb8c5169 P c47cde25b26d48d5820a6459f7b03d2a [term 1 FOLLOWER]: Refusing update from remote peer 2b1f70ba4edc4fe4ba1e2fa7e54c9638: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:24:58.294122 16558 raft_consensus.cc:1261] T 3ef84f5de382449a804bdc9ecb8c5169 P 3dc3dafa5ce240fdbfa093cf031e9ebf [term 1 FOLLOWER]: Refusing update from remote peer 2b1f70ba4edc4fe4ba1e2fa7e54c9638: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:24:58.294833 16762 consensus_queue.cc:1035] T 3ef84f5de382449a804bdc9ecb8c5169 P 2b1f70ba4edc4fe4ba1e2fa7e54c9638 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dc3dafa5ce240fdbfa093cf031e9ebf" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 46215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:24:58.295071 16764 consensus_queue.cc:1035] T 3ef84f5de382449a804bdc9ecb8c5169 P 2b1f70ba4edc4fe4ba1e2fa7e54c9638 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c47cde25b26d48d5820a6459f7b03d2a" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 39275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:24:58.302439 16768 mvcc.cc:205] Tried to move back new op lower bound from 7332594885805203456 to 7332594885677703168. Current Snapshot: MvccSnapshot[applied={T|T < 7332594885805203456}]
I20260923 17:24:58.303059 16769 mvcc.cc:205] Tried to move back new op lower bound from 7332594885805203456 to 7332594885677703168. Current Snapshot: MvccSnapshot[applied={T|T < 7332594885805203456}]
I20260923 17:24:58.311045 16281 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:24:58.311266 16281 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:24:58.313664 16281 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 93d6637c56454a1c80d723992f5fb834: Sending DeleteTablet for 3 replicas of tablet 3ef84f5de382449a804bdc9ecb8c5169
I20260923 17:24:58.314459 16405 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ef84f5de382449a804bdc9ecb8c5169 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:24:58 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:43892
I20260923 17:24:58.314543 16538 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ef84f5de382449a804bdc9ecb8c5169 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:24:58 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:59012
I20260923 17:24:58.314590 16671 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ef84f5de382449a804bdc9ecb8c5169 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:24:58 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:60986
I20260923 17:24:58.314999 16777 tablet_replica.cc:330] T 3ef84f5de382449a804bdc9ecb8c5169 P c47cde25b26d48d5820a6459f7b03d2a: stopping tablet replica
I20260923 17:24:58.315330 16777 raft_consensus.cc:2229] T 3ef84f5de382449a804bdc9ecb8c5169 P c47cde25b26d48d5820a6459f7b03d2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:24:58.315653 16777 raft_consensus.cc:2258] T 3ef84f5de382449a804bdc9ecb8c5169 P c47cde25b26d48d5820a6459f7b03d2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:24:58.315655 16778 tablet_replica.cc:330] T 3ef84f5de382449a804bdc9ecb8c5169 P 3dc3dafa5ce240fdbfa093cf031e9ebf: stopping tablet replica
I20260923 17:24:58.315697 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 16343
I20260923 17:24:58.316815 16778 raft_consensus.cc:2229] T 3ef84f5de382449a804bdc9ecb8c5169 P 3dc3dafa5ce240fdbfa093cf031e9ebf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:24:58.317135 16778 raft_consensus.cc:2258] T 3ef84f5de382449a804bdc9ecb8c5169 P 3dc3dafa5ce240fdbfa093cf031e9ebf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:24:58.318724 16778 ts_tablet_manager.cc:1905] T 3ef84f5de382449a804bdc9ecb8c5169 P 3dc3dafa5ce240fdbfa093cf031e9ebf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:24:58.315035 16779 tablet_replica.cc:330] T 3ef84f5de382449a804bdc9ecb8c5169 P 2b1f70ba4edc4fe4ba1e2fa7e54c9638: stopping tablet replica
I20260923 17:24:58.322440 16778 ts_tablet_manager.cc:1918] T 3ef84f5de382449a804bdc9ecb8c5169 P 3dc3dafa5ce240fdbfa093cf031e9ebf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:24:58.322441 16779 raft_consensus.cc:2229] T 3ef84f5de382449a804bdc9ecb8c5169 P 2b1f70ba4edc4fe4ba1e2fa7e54c9638 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:24:58.322621 16778 log.cc:1196] T 3ef84f5de382449a804bdc9ecb8c5169 P 3dc3dafa5ce240fdbfa093cf031e9ebf: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-1/wal/wals/3ef84f5de382449a804bdc9ecb8c5169
I20260923 17:24:58.322819 16779 raft_consensus.cc:2258] T 3ef84f5de382449a804bdc9ecb8c5169 P 2b1f70ba4edc4fe4ba1e2fa7e54c9638 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:24:58.323117 16778 ts_tablet_manager.cc:1939] T 3ef84f5de382449a804bdc9ecb8c5169 P 3dc3dafa5ce240fdbfa093cf031e9ebf: Deleting consensus metadata
I20260923 17:24:58.324306 16779 ts_tablet_manager.cc:1905] T 3ef84f5de382449a804bdc9ecb8c5169 P 2b1f70ba4edc4fe4ba1e2fa7e54c9638: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:24:58.324499 16268 catalog_manager.cc:4841] TS 3dc3dafa5ce240fdbfa093cf031e9ebf (127.11.229.2:46215): tablet 3ef84f5de382449a804bdc9ecb8c5169 (table test-table [id=170a3d16b2d44351a17a5631afd2c457]) successfully deleted
I20260923 17:24:58.325392 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 16476
W20260923 17:24:58.325373 16265 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.229.1:39275 (error 108)
I20260923 17:24:58.327502 16779 ts_tablet_manager.cc:1918] T 3ef84f5de382449a804bdc9ecb8c5169 P 2b1f70ba4edc4fe4ba1e2fa7e54c9638: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:24:58.327595 16779 log.cc:1196] T 3ef84f5de382449a804bdc9ecb8c5169 P 2b1f70ba4edc4fe4ba1e2fa7e54c9638: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184271597484-12180-0/minicluster-data/ts-2/wal/wals/3ef84f5de382449a804bdc9ecb8c5169
I20260923 17:24:58.327927 16779 ts_tablet_manager.cc:1939] T 3ef84f5de382449a804bdc9ecb8c5169 P 2b1f70ba4edc4fe4ba1e2fa7e54c9638: Deleting consensus metadata
I20260923 17:24:58.329106 16268 catalog_manager.cc:4841] TS 2b1f70ba4edc4fe4ba1e2fa7e54c9638 (127.11.229.3:37715): tablet 3ef84f5de382449a804bdc9ecb8c5169 (table test-table [id=170a3d16b2d44351a17a5631afd2c457]) successfully deleted
I20260923 17:24:58.329778 16748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:58.329424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:39275 (local address 127.0.0.1:43902)
0923 17:24:58.329584 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:58.329671 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.1:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260923 17:24:58.329990 16265 catalog_manager.cc:4584] TS c47cde25b26d48d5820a6459f7b03d2a (127.11.229.1:39275): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3ef84f5de382449a804bdc9ecb8c5169: Network error: Client connection negotiation failed: client connection to 127.11.229.1:39275: connect: Connection refused (error 111)
I20260923 17:24:58.332932 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 16609
I20260923 17:24:58.338487 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 16250
2026-09-23T17:24:58Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2299 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-23T17:24:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:24:58Z Disabled control of system clock
I20260923 17:24:58.369572 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:36915
--webserver_interface=127.11.229.62
--webserver_port=0
--builtin_ntp_servers=127.11.229.20:41407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:36915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 17:24:58.485351 16784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:58.485631 16784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:58.485669 16784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:58.489790 16784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:24:58.489843 16784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:58.489859 16784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:58.489876 16784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:24:58.489892 16784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:24:58.494740 16784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:41407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:36915
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:36915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:58.495913 16784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:58.497299 16784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:58.505842 16789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:58.505787 16792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:58.505895 16784 server_base.cc:1034] running on GCE node
W20260923 17:24:58.505899 16790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:58.506735 16784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:58.507895 16784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:58.509096 16784 hybrid_clock.cc:648] HybridClock initialized: now 1790184298509082 us; error 27 us; skew 500 ppm
I20260923 17:24:58.515023 16784 webserver.cc:460] Webserver started at http://127.11.229.62:42833/ using document root <none> and password file <none>
I20260923 17:24:58.515654 16784 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:58.515753 16784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:58.515995 16784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:58.518159 16784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "6dbd540033d14f1fa67da4fd0325f694"
format_stamp: "Formatted at 2026-09-23 17:24:58 on dist-test-slave-ptl9"
server_key: "2c9e3c3b52faee49bdd1330b8b905b7f"
server_key_iv: "4731174636644380efb87250554ce5bc"
server_key_version: "encryptionkey@0"
I20260923 17:24:58.518734 16784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "6dbd540033d14f1fa67da4fd0325f694"
format_stamp: "Formatted at 2026-09-23 17:24:58 on dist-test-slave-ptl9"
server_key: "2c9e3c3b52faee49bdd1330b8b905b7f"
server_key_iv: "4731174636644380efb87250554ce5bc"
server_key_version: "encryptionkey@0"
I20260923 17:24:58.522445 16784 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:24:58.524820 16798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:58.526165 16784 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:24:58.526307 16784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "6dbd540033d14f1fa67da4fd0325f694"
format_stamp: "Formatted at 2026-09-23 17:24:58 on dist-test-slave-ptl9"
server_key: "2c9e3c3b52faee49bdd1330b8b905b7f"
server_key_iv: "4731174636644380efb87250554ce5bc"
server_key_version: "encryptionkey@0"
I20260923 17:24:58.526422 16784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:58.531016 16784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:58.531618 16784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:58.531795 16784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:58.538476 16784 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:36915
I20260923 17:24:58.538520 16850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:36915 every 8 connection(s)
I20260923 17:24:58.539363 16784 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:24:58.542460 16851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:58.544914 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 16784
I20260923 17:24:58.545037 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:24:58.545368 12180 external_mini_cluster.cc:1428] Setting key 06b4161178d0c46397fb1921a1ba7155
I20260923 17:24:58.548354 16851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dbd540033d14f1fa67da4fd0325f694: Bootstrap starting.
I20260923 17:24:58.550796 16854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:58.546777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52430 (local address 127.11.229.62:36915)
0923 17:24:58.547387 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:58.547402 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:58.547437 (+ 35us) server_negotiation.cc:407] Connection header received
0923 17:24:58.548046 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:58.548073 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:58.548447 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:58.548746 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:58.549543 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:58.549548 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:58.549561 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:24:58.549571 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:58.549621 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:24:58.549636 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:58.549744 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:58.549844 (+ 100us) server_negotiation.cc:299] Negotiation successful
0923 17:24:58.550138 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":133,"threads_started":1}
I20260923 17:24:58.550980 16851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6dbd540033d14f1fa67da4fd0325f694: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:58.551688 16851 log.cc:824] T 00000000000000000000000000000000 P 6dbd540033d14f1fa67da4fd0325f694: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:58.553692 16851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dbd540033d14f1fa67da4fd0325f694: No bootstrap required, opened a new log
I20260923 17:24:58.557078 16851 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6dbd540033d14f1fa67da4fd0325f694 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbd540033d14f1fa67da4fd0325f694" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 36915 } }
I20260923 17:24:58.557350 16851 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6dbd540033d14f1fa67da4fd0325f694 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:58.557397 16851 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6dbd540033d14f1fa67da4fd0325f694 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dbd540033d14f1fa67da4fd0325f694, State: Initialized, Role: FOLLOWER
I20260923 17:24:58.557753 16851 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6dbd540033d14f1fa67da4fd0325f694 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbd540033d14f1fa67da4fd0325f694" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 36915 } }
I20260923 17:24:58.557863 16851 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6dbd540033d14f1fa67da4fd0325f694 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:24:58.557905 16851 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6dbd540033d14f1fa67da4fd0325f694 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:24:58.557972 16851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6dbd540033d14f1fa67da4fd0325f694 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:58.558771 16851 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6dbd540033d14f1fa67da4fd0325f694 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbd540033d14f1fa67da4fd0325f694" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 36915 } }
I20260923 17:24:58.559048 16851 leader_election.cc:302] T 00000000000000000000000000000000 P 6dbd540033d14f1fa67da4fd0325f694 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6dbd540033d14f1fa67da4fd0325f694; no voters:
I20260923 17:24:58.559286 16851 leader_election.cc:288] T 00000000000000000000000000000000 P 6dbd540033d14f1fa67da4fd0325f694 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:24:58.559430 16856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6dbd540033d14f1fa67da4fd0325f694 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:24:58.559703 16856 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6dbd540033d14f1fa67da4fd0325f694 [term 1 LEADER]: Becoming Leader. State: Replica: 6dbd540033d14f1fa67da4fd0325f694, State: Running, Role: LEADER
I20260923 17:24:58.560129 16856 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6dbd540033d14f1fa67da4fd0325f694 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbd540033d14f1fa67da4fd0325f694" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 36915 } }
I20260923 17:24:58.560503 16851 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6dbd540033d14f1fa67da4fd0325f694 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:24:58.561937 16858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dbd540033d14f1fa67da4fd0325f694 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dbd540033d14f1fa67da4fd0325f694. Latest consensus state: current_term: 1 leader_uuid: "6dbd540033d14f1fa67da4fd0325f694" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbd540033d14f1fa67da4fd0325f694" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 36915 } } }
I20260923 17:24:58.562103 16858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dbd540033d14f1fa67da4fd0325f694 [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:58.562520 16865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:24:58.562489 16857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dbd540033d14f1fa67da4fd0325f694 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6dbd540033d14f1fa67da4fd0325f694" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbd540033d14f1fa67da4fd0325f694" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 36915 } } }
I20260923 17:24:58.562629 16857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dbd540033d14f1fa67da4fd0325f694 [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:58.565857 16865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:24:58.571673 16865 catalog_manager.cc:1347] Generated new cluster ID: 08576e8f6bb94dbf883f139d0099f6aa
I20260923 17:24:58.571756 16865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:24:58.615733 16865 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:24:58.616113 16865 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:24:58.622056 16865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6dbd540033d14f1fa67da4fd0325f694: Generated new TSK 0
I20260923 17:24:58.622886 16865 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:24:58.626777 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.1:0
--local_ip_for_outbound_sockets=127.11.229.1
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:36915
--builtin_ntp_servers=127.11.229.20:41407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 17:24:58.739805 16875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:58.740057 16875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:58.740095 16875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:58.744065 16875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:58.744117 16875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:58.744168 16875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.1
I20260923 17:24:58.748765 16875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:41407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:36915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.1
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:58.749908 16875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:58.751302 16875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:58.760887 16881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:58.760874 16880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:58.760874 16883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:58.761166 16875 server_base.cc:1034] running on GCE node
I20260923 17:24:58.761675 16875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:58.762462 16875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:58.763626 16875 hybrid_clock.cc:648] HybridClock initialized: now 1790184298763607 us; error 29 us; skew 500 ppm
I20260923 17:24:58.765686 16875 webserver.cc:460] Webserver started at http://127.11.229.1:44957/ using document root <none> and password file <none>
I20260923 17:24:58.766294 16875 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:58.766357 16875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:58.766567 16875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:58.768684 16875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-0/data/instance:
uuid: "05272858971e4fa4a3509a4374e60c6c"
format_stamp: "Formatted at 2026-09-23 17:24:58 on dist-test-slave-ptl9"
server_key: "5d69de7b72e7b1e8d3c6cb3bc6d9ef3a"
server_key_iv: "cfe11b58cbab4b8bdc0fd0625f011c06"
server_key_version: "encryptionkey@0"
I20260923 17:24:58.769320 16875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance:
uuid: "05272858971e4fa4a3509a4374e60c6c"
format_stamp: "Formatted at 2026-09-23 17:24:58 on dist-test-slave-ptl9"
server_key: "5d69de7b72e7b1e8d3c6cb3bc6d9ef3a"
server_key_iv: "cfe11b58cbab4b8bdc0fd0625f011c06"
server_key_version: "encryptionkey@0"
I20260923 17:24:58.772979 16875 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:24:58.775574 16889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:58.776793 16875 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260923 17:24:58.776927 16875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-0/wal
uuid: "05272858971e4fa4a3509a4374e60c6c"
format_stamp: "Formatted at 2026-09-23 17:24:58 on dist-test-slave-ptl9"
server_key: "5d69de7b72e7b1e8d3c6cb3bc6d9ef3a"
server_key_iv: "cfe11b58cbab4b8bdc0fd0625f011c06"
server_key_version: "encryptionkey@0"
I20260923 17:24:58.777066 16875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:58.781700 16875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:58.782321 16875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:58.782503 16875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:58.783102 16875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:24:58.784005 16875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:24:58.784083 16875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:58.784148 16875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:24:58.784216 16875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:58.792986 16875 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.1:40839
I20260923 17:24:58.793028 17002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.1:40839 every 8 connection(s)
I20260923 17:24:58.793979 16875 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
I20260923 17:24:58.800148 16854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:58.795905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:35015 (local address 127.11.229.62:36915)
0923 17:24:58.796042 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:58.796046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:58.797133 (+ 1087us) server_negotiation.cc:407] Connection header received
0923 17:24:58.798563 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:58.798566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:58.798604 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:58.798677 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:58.799796 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:58.799798 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:58.799800 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:24:58.799809 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:58.799823 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:24:58.799830 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:58.799898 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:58.800000 (+ 102us) server_negotiation.cc:299] Negotiation successful
0923 17:24:58.800036 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 17:24:58.801028 17005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:58.796296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:36915 (local address 127.11.229.1:35015)
0923 17:24:58.796978 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:58.797017 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:58.798377 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:58.798842 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:58.798849 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:58.798938 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:58.799546 (+ 608us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:24:58.799557 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:58.799599 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:24:58.799667 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:58.799897 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:58.799901 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:58.799905 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 17:24:58.800028 (+ 123us) client_negotiation.cc:240] Negotiation successful
0923 17:24:58.800254 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":109,"threads_started":1}
I20260923 17:24:58.802310 17003 heartbeater.cc:347] Connected to a master server at 127.11.229.62:36915
I20260923 17:24:58.802515 17003 heartbeater.cc:464] Registering TS with master...
I20260923 17:24:58.802886 17003 heartbeater.cc:511] Master 127.11.229.62:36915 requested a full tablet report, sending...
I20260923 17:24:58.803355 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 16875
I20260923 17:24:58.803506 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance
I20260923 17:24:58.803843 12180 external_mini_cluster.cc:1428] Setting key 7743f45158cd9bc2f9ece111ecf3c510
I20260923 17:24:58.804637 16815 ts_manager.cc:194] Registered new tserver with Master: 05272858971e4fa4a3509a4374e60c6c (127.11.229.1:40839)
I20260923 17:24:58.806037 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.2:0
--local_ip_for_outbound_sockets=127.11.229.2
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:36915
--builtin_ntp_servers=127.11.229.20:41407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 17:24:58.918175 17006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:58.918460 17006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:58.918533 17006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:58.922538 17006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:58.922619 17006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:58.922711 17006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.2
I20260923 17:24:58.927501 17006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:41407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:36915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.2
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:58.928802 17006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:58.930346 17006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:58.940090 17011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:58.940165 17012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:58.940223 17006 server_base.cc:1034] running on GCE node
W20260923 17:24:58.940434 17014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:58.940862 17006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:58.941560 17006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:58.942720 17006 hybrid_clock.cc:648] HybridClock initialized: now 1790184298942710 us; error 34 us; skew 500 ppm
I20260923 17:24:58.944649 17006 webserver.cc:460] Webserver started at http://127.11.229.2:42513/ using document root <none> and password file <none>
I20260923 17:24:58.945254 17006 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:58.945377 17006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:58.945618 17006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:58.947731 17006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-1/data/instance:
uuid: "1bd67000226647f38f7dc0ea3e9ebb0e"
format_stamp: "Formatted at 2026-09-23 17:24:58 on dist-test-slave-ptl9"
server_key: "3f494411655e25ef6a9b5ab9c60ceffa"
server_key_iv: "c2eea1c1ebeceeb49629b4a509fbd570"
server_key_version: "encryptionkey@0"
I20260923 17:24:58.948305 17006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance:
uuid: "1bd67000226647f38f7dc0ea3e9ebb0e"
format_stamp: "Formatted at 2026-09-23 17:24:58 on dist-test-slave-ptl9"
server_key: "3f494411655e25ef6a9b5ab9c60ceffa"
server_key_iv: "c2eea1c1ebeceeb49629b4a509fbd570"
server_key_version: "encryptionkey@0"
I20260923 17:24:58.952052 17006 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260923 17:24:58.954648 17020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:58.955840 17006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:24:58.955964 17006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-1/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-1/wal
uuid: "1bd67000226647f38f7dc0ea3e9ebb0e"
format_stamp: "Formatted at 2026-09-23 17:24:58 on dist-test-slave-ptl9"
server_key: "3f494411655e25ef6a9b5ab9c60ceffa"
server_key_iv: "c2eea1c1ebeceeb49629b4a509fbd570"
server_key_version: "encryptionkey@0"
I20260923 17:24:58.956079 17006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:58.960711 17006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:58.961421 17006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:58.961635 17006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:58.962287 17006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:24:58.963217 17006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:24:58.963263 17006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:58.963326 17006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:24:58.963359 17006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:58.971973 17006 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.2:37785
I20260923 17:24:58.972000 17133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.2:37785 every 8 connection(s)
I20260923 17:24:58.972873 17006 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
I20260923 17:24:58.978782 16854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:58.974754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:57725 (local address 127.11.229.62:36915)
0923 17:24:58.974917 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:58.974921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:58.975987 (+ 1066us) server_negotiation.cc:407] Connection header received
0923 17:24:58.977239 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:58.977242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:58.977323 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:58.977404 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:58.978419 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:58.978422 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:58.978424 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:24:58.978435 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:58.978454 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:24:58.978461 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:58.978533 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:58.978637 (+ 104us) server_negotiation.cc:299] Negotiation successful
0923 17:24:58.978675 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 17:24:58.979625 17136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:58.975179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:36915 (local address 127.11.229.2:57725)
0923 17:24:58.975841 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:58.975881 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:58.977042 (+ 1161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:58.977577 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:58.977585 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:58.977676 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:58.978176 (+ 500us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:24:58.978177 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:58.978217 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:24:58.978284 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:58.978521 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:58.978524 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:58.978528 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 17:24:58.978656 (+ 128us) client_negotiation.cc:240] Negotiation successful
0923 17:24:58.978843 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":113,"threads_started":1}
I20260923 17:24:58.980707 17134 heartbeater.cc:347] Connected to a master server at 127.11.229.62:36915
I20260923 17:24:58.980958 17134 heartbeater.cc:464] Registering TS with master...
I20260923 17:24:58.981400 17134 heartbeater.cc:511] Master 127.11.229.62:36915 requested a full tablet report, sending...
I20260923 17:24:58.982414 16815 ts_manager.cc:194] Registered new tserver with Master: 1bd67000226647f38f7dc0ea3e9ebb0e (127.11.229.2:37785)
I20260923 17:24:58.982430 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 17006
I20260923 17:24:58.982546 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance
I20260923 17:24:58.982865 12180 external_mini_cluster.cc:1428] Setting key 15636e3b4f740fc540b17093ec26c5d0
I20260923 17:24:58.984733 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.3:0
--local_ip_for_outbound_sockets=127.11.229.3
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:36915
--builtin_ntp_servers=127.11.229.20:41407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 17:24:59.094811 17137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:59.095063 17137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:59.095101 17137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:59.099126 17137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:59.099179 17137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:59.099231 17137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.3
I20260923 17:24:59.103762 17137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:41407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:36915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.3
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:59.104751 17137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:59.106060 17137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:59.115569 17143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:59.115648 17137 server_base.cc:1034] running on GCE node
W20260923 17:24:59.115687 17145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:59.115885 17142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:59.116302 17137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:59.116945 17137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:59.118117 17137 hybrid_clock.cc:648] HybridClock initialized: now 1790184299118104 us; error 33 us; skew 500 ppm
I20260923 17:24:59.119933 17137 webserver.cc:460] Webserver started at http://127.11.229.3:35319/ using document root <none> and password file <none>
I20260923 17:24:59.120528 17137 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:59.120594 17137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:59.120832 17137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:59.123119 17137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-2/data/instance:
uuid: "8ca00a23463c43428602e86e93a085c2"
format_stamp: "Formatted at 2026-09-23 17:24:59 on dist-test-slave-ptl9"
server_key: "68a0ce38aba6769c9b9a959fde82edfc"
server_key_iv: "9361b30c330ef3f0cc540cd0d746ca24"
server_key_version: "encryptionkey@0"
I20260923 17:24:59.123682 17137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance:
uuid: "8ca00a23463c43428602e86e93a085c2"
format_stamp: "Formatted at 2026-09-23 17:24:59 on dist-test-slave-ptl9"
server_key: "68a0ce38aba6769c9b9a959fde82edfc"
server_key_iv: "9361b30c330ef3f0cc540cd0d746ca24"
server_key_version: "encryptionkey@0"
I20260923 17:24:59.127372 17137 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260923 17:24:59.129930 17151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:59.131220 17137 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:24:59.131353 17137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-2/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-2/wal
uuid: "8ca00a23463c43428602e86e93a085c2"
format_stamp: "Formatted at 2026-09-23 17:24:59 on dist-test-slave-ptl9"
server_key: "68a0ce38aba6769c9b9a959fde82edfc"
server_key_iv: "9361b30c330ef3f0cc540cd0d746ca24"
server_key_version: "encryptionkey@0"
I20260923 17:24:59.131456 17137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:59.136010 17137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:59.136632 17137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:59.136803 17137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:59.137390 17137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:24:59.138299 17137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:24:59.138345 17137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:59.138407 17137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:24:59.138443 17137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:59.146788 17137 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.3:34021
I20260923 17:24:59.146854 17264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.3:34021 every 8 connection(s)
I20260923 17:24:59.147708 17137 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
I20260923 17:24:59.150354 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 17137
I20260923 17:24:59.150481 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance
I20260923 17:24:59.150770 12180 external_mini_cluster.cc:1428] Setting key 428ae412818c5cb6b1b0bfb5f4a8c7d6
I20260923 17:24:59.154247 16854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:59.149583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:37761 (local address 127.11.229.62:36915)
0923 17:24:59.149717 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:59.149721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:59.150951 (+ 1230us) server_negotiation.cc:407] Connection header received
0923 17:24:59.152656 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:59.152658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:59.152694 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:59.152793 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:59.153884 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:59.153887 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:59.153889 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:24:59.153901 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:59.153921 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:24:59.153929 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:59.154004 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:59.154105 (+ 101us) server_negotiation.cc:299] Negotiation successful
0923 17:24:59.154146 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 17:24:59.155112 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:59.150007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:36915 (local address 127.11.229.3:37761)
0923 17:24:59.150784 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:59.150834 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:59.152425 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:59.152991 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:59.153000 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:59.153106 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:59.153641 (+ 535us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:24:59.153643 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:59.153683 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:24:59.153752 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:59.153994 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:59.153998 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:59.154002 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 17:24:59.154125 (+ 123us) client_negotiation.cc:240] Negotiation successful
0923 17:24:59.154341 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":152,"threads_started":1}
I20260923 17:24:59.156172 17265 heartbeater.cc:347] Connected to a master server at 127.11.229.62:36915
I20260923 17:24:59.156404 17265 heartbeater.cc:464] Registering TS with master...
I20260923 17:24:59.156813 17265 heartbeater.cc:511] Master 127.11.229.62:36915 requested a full tablet report, sending...
I20260923 17:24:59.157841 16815 ts_manager.cc:194] Registered new tserver with Master: 8ca00a23463c43428602e86e93a085c2 (127.11.229.3:34021)
I20260923 17:24:59.164623 12180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:24:59.167897 16854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:59.166525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52444 (local address 127.11.229.62:36915)
0923 17:24:59.166649 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:59.166652 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:59.166775 (+ 123us) server_negotiation.cc:407] Connection header received
0923 17:24:59.166859 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:59.166861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:59.166892 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:59.166957 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:59.167583 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:59.167586 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:59.167588 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:24:59.167599 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:59.167615 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:24:59.167623 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:59.167692 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:59.167767 (+ 75us) server_negotiation.cc:299] Negotiation successful
0923 17:24:59.167798 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 17:24:59.170715 16815 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52444:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:24:59.173763 16815 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:24:59.185724 17279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:59.182970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55634 (local address 127.11.229.1:40839)
0923 17:24:59.183610 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:59.183623 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:59.183651 (+ 28us) server_negotiation.cc:407] Connection header received
0923 17:24:59.183737 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:59.183743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:59.183949 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:59.184141 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:59.185053 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:59.185063 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:59.185079 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:24:59.185094 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:59.185154 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:24:59.185190 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:59.185335 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:59.185437 (+ 102us) server_negotiation.cc:299] Negotiation successful
0923 17:24:59.185534 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":156,"threads_started":1}
I20260923 17:24:59.185907 17276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:59.182427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:40839 (local address 127.0.0.1:55634)
0923 17:24:59.183446 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:59.183497 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:59.183601 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:59.184153 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:59.184160 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:59.184245 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:59.184799 (+ 554us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:24:59.184801 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:59.184822 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:24:59.184893 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:59.185292 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:59.185302 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:59.185309 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 17:24:59.185424 (+ 115us) client_negotiation.cc:240] Negotiation successful
0923 17:24:59.185491 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":867,"thread_start_us":111,"threads_started":1}
I20260923 17:24:59.186761 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:59.183080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:37785 (local address 127.0.0.1:52676)
0923 17:24:59.183867 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:59.183881 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:59.183962 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:59.185404 (+ 1442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:59.185407 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:59.185422 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:59.185941 (+ 519us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:24:59.185943 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:59.185955 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:24:59.186043 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:59.186432 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:59.186435 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:59.186438 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 17:24:59.186586 (+ 148us) client_negotiation.cc:240] Negotiation successful
0923 17:24:59.186627 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":76,"threads_started":1}
I20260923 17:24:59.186761 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:59.182332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:34021 (local address 127.0.0.1:49470)
0923 17:24:59.183745 (+ 1413us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:59.183758 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:59.183817 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:59.184772 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:59.184775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:59.184790 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:59.185224 (+ 434us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:24:59.185226 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:59.185237 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:24:59.185292 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:59.186558 (+ 1266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:59.186564 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:59.186568 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 17:24:59.186647 (+ 79us) client_negotiation.cc:240] Negotiation successful
0923 17:24:59.186671 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":65,"threads_started":1}
I20260923 17:24:59.187126 17281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:59.183874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52676 (local address 127.11.229.2:37785)
0923 17:24:59.184320 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:59.184332 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:59.184355 (+ 23us) server_negotiation.cc:407] Connection header received
0923 17:24:59.184425 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:59.184433 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:59.184602 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:59.184752 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:59.186181 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:59.186187 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:59.186199 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:24:59.186211 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:59.186253 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:24:59.186284 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:59.186389 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:59.186934 (+ 545us) server_negotiation.cc:299] Negotiation successful
0923 17:24:59.187009 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":118,"threads_started":1}
I20260923 17:24:59.187400 17278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:59.182442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49470 (local address 127.11.229.3:34021)
0923 17:24:59.184062 (+ 1620us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:59.184078 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:59.184101 (+ 23us) server_negotiation.cc:407] Connection header received
0923 17:24:59.184150 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:59.184155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:59.184316 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:59.185020 (+ 704us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:59.186391 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:59.186396 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:59.186407 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:24:59.186418 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:59.186456 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:24:59.186479 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:59.186582 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:59.186685 (+ 103us) server_negotiation.cc:299] Negotiation successful
0923 17:24:59.186752 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1502,"thread_start_us":117,"threads_started":1}
I20260923 17:24:59.189009 16937 tablet_service.cc:1467] Processing CreateTablet for tablet 6dff6a7601394e41b5ce523f1a243643 (DEFAULT_TABLE table=test-table [id=100e80b78f2a4d879d8b625f859b87fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:24:59.190162 16937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dff6a7601394e41b5ce523f1a243643. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:59.189903 17068 tablet_service.cc:1467] Processing CreateTablet for tablet 6dff6a7601394e41b5ce523f1a243643 (DEFAULT_TABLE table=test-table [id=100e80b78f2a4d879d8b625f859b87fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:24:59.189903 17199 tablet_service.cc:1467] Processing CreateTablet for tablet 6dff6a7601394e41b5ce523f1a243643 (DEFAULT_TABLE table=test-table [id=100e80b78f2a4d879d8b625f859b87fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:24:59.190917 17068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dff6a7601394e41b5ce523f1a243643. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:59.190917 17199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dff6a7601394e41b5ce523f1a243643. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:59.196372 17282 tablet_bootstrap.cc:492] T 6dff6a7601394e41b5ce523f1a243643 P 05272858971e4fa4a3509a4374e60c6c: Bootstrap starting.
I20260923 17:24:59.198084 17283 tablet_bootstrap.cc:492] T 6dff6a7601394e41b5ce523f1a243643 P 1bd67000226647f38f7dc0ea3e9ebb0e: Bootstrap starting.
I20260923 17:24:59.199762 17282 tablet_bootstrap.cc:654] T 6dff6a7601394e41b5ce523f1a243643 P 05272858971e4fa4a3509a4374e60c6c: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:59.200341 17283 tablet_bootstrap.cc:654] T 6dff6a7601394e41b5ce523f1a243643 P 1bd67000226647f38f7dc0ea3e9ebb0e: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:59.200707 17282 log.cc:824] T 6dff6a7601394e41b5ce523f1a243643 P 05272858971e4fa4a3509a4374e60c6c: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:59.201164 17283 log.cc:824] T 6dff6a7601394e41b5ce523f1a243643 P 1bd67000226647f38f7dc0ea3e9ebb0e: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:59.201443 17284 tablet_bootstrap.cc:492] T 6dff6a7601394e41b5ce523f1a243643 P 8ca00a23463c43428602e86e93a085c2: Bootstrap starting.
I20260923 17:24:59.202773 17282 tablet_bootstrap.cc:492] T 6dff6a7601394e41b5ce523f1a243643 P 05272858971e4fa4a3509a4374e60c6c: No bootstrap required, opened a new log
I20260923 17:24:59.202996 17282 ts_tablet_manager.cc:1397] T 6dff6a7601394e41b5ce523f1a243643 P 05272858971e4fa4a3509a4374e60c6c: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260923 17:24:59.203235 17283 tablet_bootstrap.cc:492] T 6dff6a7601394e41b5ce523f1a243643 P 1bd67000226647f38f7dc0ea3e9ebb0e: No bootstrap required, opened a new log
I20260923 17:24:59.203418 17283 ts_tablet_manager.cc:1397] T 6dff6a7601394e41b5ce523f1a243643 P 1bd67000226647f38f7dc0ea3e9ebb0e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 17:24:59.203596 17284 tablet_bootstrap.cc:654] T 6dff6a7601394e41b5ce523f1a243643 P 8ca00a23463c43428602e86e93a085c2: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:59.204391 17284 log.cc:824] T 6dff6a7601394e41b5ce523f1a243643 P 8ca00a23463c43428602e86e93a085c2: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:59.206490 17284 tablet_bootstrap.cc:492] T 6dff6a7601394e41b5ce523f1a243643 P 8ca00a23463c43428602e86e93a085c2: No bootstrap required, opened a new log
I20260923 17:24:59.206697 17284 ts_tablet_manager.cc:1397] T 6dff6a7601394e41b5ce523f1a243643 P 8ca00a23463c43428602e86e93a085c2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 17:24:59.207280 17282 raft_consensus.cc:348] T 6dff6a7601394e41b5ce523f1a243643 P 05272858971e4fa4a3509a4374e60c6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca00a23463c43428602e86e93a085c2" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34021 } } peers { permanent_uuid: "05272858971e4fa4a3509a4374e60c6c" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 40839 } } peers { permanent_uuid: "1bd67000226647f38f7dc0ea3e9ebb0e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 37785 } }
I20260923 17:24:59.207571 17282 raft_consensus.cc:374] T 6dff6a7601394e41b5ce523f1a243643 P 05272858971e4fa4a3509a4374e60c6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:59.207626 17282 raft_consensus.cc:729] T 6dff6a7601394e41b5ce523f1a243643 P 05272858971e4fa4a3509a4374e60c6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05272858971e4fa4a3509a4374e60c6c, State: Initialized, Role: FOLLOWER
I20260923 17:24:59.207758 17283 raft_consensus.cc:348] T 6dff6a7601394e41b5ce523f1a243643 P 1bd67000226647f38f7dc0ea3e9ebb0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca00a23463c43428602e86e93a085c2" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34021 } } peers { permanent_uuid: "05272858971e4fa4a3509a4374e60c6c" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 40839 } } peers { permanent_uuid: "1bd67000226647f38f7dc0ea3e9ebb0e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 37785 } }
I20260923 17:24:59.207999 17283 raft_consensus.cc:374] T 6dff6a7601394e41b5ce523f1a243643 P 1bd67000226647f38f7dc0ea3e9ebb0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:59.208048 17283 raft_consensus.cc:729] T 6dff6a7601394e41b5ce523f1a243643 P 1bd67000226647f38f7dc0ea3e9ebb0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bd67000226647f38f7dc0ea3e9ebb0e, State: Initialized, Role: FOLLOWER
I20260923 17:24:59.208127 17282 consensus_queue.cc:260] T 6dff6a7601394e41b5ce523f1a243643 P 05272858971e4fa4a3509a4374e60c6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca00a23463c43428602e86e93a085c2" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34021 } } peers { permanent_uuid: "05272858971e4fa4a3509a4374e60c6c" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 40839 } } peers { permanent_uuid: "1bd67000226647f38f7dc0ea3e9ebb0e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 37785 } }
I20260923 17:24:59.208490 17283 consensus_queue.cc:260] T 6dff6a7601394e41b5ce523f1a243643 P 1bd67000226647f38f7dc0ea3e9ebb0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca00a23463c43428602e86e93a085c2" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34021 } } peers { permanent_uuid: "05272858971e4fa4a3509a4374e60c6c" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 40839 } } peers { permanent_uuid: "1bd67000226647f38f7dc0ea3e9ebb0e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 37785 } }
I20260923 17:24:59.208890 17003 heartbeater.cc:503] Master 127.11.229.62:36915 was elected leader, sending a full tablet report...
I20260923 17:24:59.209172 17282 ts_tablet_manager.cc:1428] T 6dff6a7601394e41b5ce523f1a243643 P 05272858971e4fa4a3509a4374e60c6c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260923 17:24:59.209434 17283 ts_tablet_manager.cc:1428] T 6dff6a7601394e41b5ce523f1a243643 P 1bd67000226647f38f7dc0ea3e9ebb0e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260923 17:24:59.209753 17134 heartbeater.cc:503] Master 127.11.229.62:36915 was elected leader, sending a full tablet report...
I20260923 17:24:59.212872 17284 raft_consensus.cc:348] T 6dff6a7601394e41b5ce523f1a243643 P 8ca00a23463c43428602e86e93a085c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca00a23463c43428602e86e93a085c2" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34021 } } peers { permanent_uuid: "05272858971e4fa4a3509a4374e60c6c" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 40839 } } peers { permanent_uuid: "1bd67000226647f38f7dc0ea3e9ebb0e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 37785 } }
I20260923 17:24:59.213172 17284 raft_consensus.cc:374] T 6dff6a7601394e41b5ce523f1a243643 P 8ca00a23463c43428602e86e93a085c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:59.213248 17284 raft_consensus.cc:729] T 6dff6a7601394e41b5ce523f1a243643 P 8ca00a23463c43428602e86e93a085c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ca00a23463c43428602e86e93a085c2, State: Initialized, Role: FOLLOWER
I20260923 17:24:59.213739 17284 consensus_queue.cc:260] T 6dff6a7601394e41b5ce523f1a243643 P 8ca00a23463c43428602e86e93a085c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca00a23463c43428602e86e93a085c2" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34021 } } peers { permanent_uuid: "05272858971e4fa4a3509a4374e60c6c" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 40839 } } peers { permanent_uuid: "1bd67000226647f38f7dc0ea3e9ebb0e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 37785 } }
I20260923 17:24:59.214408 17265 heartbeater.cc:503] Master 127.11.229.62:36915 was elected leader, sending a full tablet report...
I20260923 17:24:59.215672 17284 ts_tablet_manager.cc:1428] T 6dff6a7601394e41b5ce523f1a243643 P 8ca00a23463c43428602e86e93a085c2: Time spent starting tablet: real 0.009s user 0.000s sys 0.007s
W20260923 17:24:59.223860 17135 tablet.cc:2369] T 6dff6a7601394e41b5ce523f1a243643 P 1bd67000226647f38f7dc0ea3e9ebb0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:24:59.289680 17288 raft_consensus.cc:482] T 6dff6a7601394e41b5ce523f1a243643 P 05272858971e4fa4a3509a4374e60c6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:24:59.289886 17288 raft_consensus.cc:504] T 6dff6a7601394e41b5ce523f1a243643 P 05272858971e4fa4a3509a4374e60c6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca00a23463c43428602e86e93a085c2" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34021 } } peers { permanent_uuid: "05272858971e4fa4a3509a4374e60c6c" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 40839 } } peers { permanent_uuid: "1bd67000226647f38f7dc0ea3e9ebb0e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 37785 } }
I20260923 17:24:59.291023 17288 leader_election.cc:288] T 6dff6a7601394e41b5ce523f1a243643 P 05272858971e4fa4a3509a4374e60c6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ca00a23463c43428602e86e93a085c2 (127.11.229.3:34021), 1bd67000226647f38f7dc0ea3e9ebb0e (127.11.229.2:37785)
I20260923 17:24:59.293000 17005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:59.291314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:34021 (local address 127.11.229.1:59869)
0923 17:24:59.291460 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:59.291478 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:59.291574 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:59.291853 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:59.291856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:59.291873 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:59.292414 (+ 541us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:24:59.292416 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:59.292435 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:24:59.292462 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:59.292735 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:59.292739 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:59.292743 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 17:24:59.292817 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 17:24:59.292854 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260923 17:24:59.293000 17278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:59.291441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:59869 (local address 127.11.229.3:34021)
0923 17:24:59.291588 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:59.291596 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:59.291615 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:24:59.291670 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:59.291674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:59.291716 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:59.291831 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:59.292587 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:59.292590 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:59.292593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:24:59.292605 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:59.292627 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:24:59.292642 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:59.292735 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:59.292815 (+ 80us) server_negotiation.cc:299] Negotiation successful
0923 17:24:59.292855 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 17:24:59.293613 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:59.291641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:37785 (local address 127.11.229.1:33113)
0923 17:24:59.292026 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:59.292037 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:59.292141 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:59.292385 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:59.292388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:59.292401 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:59.292882 (+ 481us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:24:59.292884 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:59.292895 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:24:59.292938 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:59.293203 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:59.293207 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:59.293211 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 17:24:59.293448 (+ 237us) client_negotiation.cc:240] Negotiation successful
0923 17:24:59.293484 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":106,"threads_started":1}
I20260923 17:24:59.293618 17281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:59.291788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:33113 (local address 127.11.229.2:37785)
0923 17:24:59.291921 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:59.291925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:59.292118 (+ 193us) server_negotiation.cc:407] Connection header received
0923 17:24:59.292246 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:59.292248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:59.292281 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:59.292369 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:59.293060 (+ 691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:59.293063 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:59.293066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:24:59.293078 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:59.293114 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:24:59.293124 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:59.293215 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:59.293461 (+ 246us) server_negotiation.cc:299] Negotiation successful
0923 17:24:59.293507 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 17:24:59.293709 17219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dff6a7601394e41b5ce523f1a243643" candidate_uuid: "05272858971e4fa4a3509a4374e60c6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ca00a23463c43428602e86e93a085c2" is_pre_election: true
I20260923 17:24:59.294070 17219 raft_consensus.cc:2454] T 6dff6a7601394e41b5ce523f1a243643 P 8ca00a23463c43428602e86e93a085c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05272858971e4fa4a3509a4374e60c6c in term 0.
I20260923 17:24:59.294637 16893 leader_election.cc:302] T 6dff6a7601394e41b5ce523f1a243643 P 05272858971e4fa4a3509a4374e60c6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05272858971e4fa4a3509a4374e60c6c, 8ca00a23463c43428602e86e93a085c2; no voters:
I20260923 17:24:59.294894 17088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dff6a7601394e41b5ce523f1a243643" candidate_uuid: "05272858971e4fa4a3509a4374e60c6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bd67000226647f38f7dc0ea3e9ebb0e" is_pre_election: true
I20260923 17:24:59.294952 17288 raft_consensus.cc:2781] T 6dff6a7601394e41b5ce523f1a243643 P 05272858971e4fa4a3509a4374e60c6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:24:59.295035 17288 raft_consensus.cc:482] T 6dff6a7601394e41b5ce523f1a243643 P 05272858971e4fa4a3509a4374e60c6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:24:59.295087 17288 raft_consensus.cc:3037] T 6dff6a7601394e41b5ce523f1a243643 P 05272858971e4fa4a3509a4374e60c6c [term 0 FOLLOWER]: Advancing to term 1
W20260923 17:24:59.295161 17004 tablet.cc:2369] T 6dff6a7601394e41b5ce523f1a243643 P 05272858971e4fa4a3509a4374e60c6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:24:59.295205 17088 raft_consensus.cc:2454] T 6dff6a7601394e41b5ce523f1a243643 P 1bd67000226647f38f7dc0ea3e9ebb0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05272858971e4fa4a3509a4374e60c6c in term 0.
I20260923 17:24:59.296342 17288 raft_consensus.cc:504] T 6dff6a7601394e41b5ce523f1a243643 P 05272858971e4fa4a3509a4374e60c6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca00a23463c43428602e86e93a085c2" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34021 } } peers { permanent_uuid: "05272858971e4fa4a3509a4374e60c6c" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 40839 } } peers { permanent_uuid: "1bd67000226647f38f7dc0ea3e9ebb0e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 37785 } }
I20260923 17:24:59.296700 17288 leader_election.cc:288] T 6dff6a7601394e41b5ce523f1a243643 P 05272858971e4fa4a3509a4374e60c6c [CANDIDATE]: Term 1 election: Requested vote from peers 8ca00a23463c43428602e86e93a085c2 (127.11.229.3:34021), 1bd67000226647f38f7dc0ea3e9ebb0e (127.11.229.2:37785)
I20260923 17:24:59.297101 17088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dff6a7601394e41b5ce523f1a243643" candidate_uuid: "05272858971e4fa4a3509a4374e60c6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bd67000226647f38f7dc0ea3e9ebb0e"
I20260923 17:24:59.297224 17088 raft_consensus.cc:3037] T 6dff6a7601394e41b5ce523f1a243643 P 1bd67000226647f38f7dc0ea3e9ebb0e [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:59.297212 17219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dff6a7601394e41b5ce523f1a243643" candidate_uuid: "05272858971e4fa4a3509a4374e60c6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ca00a23463c43428602e86e93a085c2"
I20260923 17:24:59.297381 17219 raft_consensus.cc:3037] T 6dff6a7601394e41b5ce523f1a243643 P 8ca00a23463c43428602e86e93a085c2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:59.298274 17088 raft_consensus.cc:2454] T 6dff6a7601394e41b5ce523f1a243643 P 1bd67000226647f38f7dc0ea3e9ebb0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05272858971e4fa4a3509a4374e60c6c in term 1.
I20260923 17:24:59.298468 17219 raft_consensus.cc:2454] T 6dff6a7601394e41b5ce523f1a243643 P 8ca00a23463c43428602e86e93a085c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05272858971e4fa4a3509a4374e60c6c in term 1.
I20260923 17:24:59.298573 16893 leader_election.cc:302] T 6dff6a7601394e41b5ce523f1a243643 P 05272858971e4fa4a3509a4374e60c6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05272858971e4fa4a3509a4374e60c6c, 1bd67000226647f38f7dc0ea3e9ebb0e; no voters:
I20260923 17:24:59.298738 17288 raft_consensus.cc:2781] T 6dff6a7601394e41b5ce523f1a243643 P 05272858971e4fa4a3509a4374e60c6c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:24:59.298970 17288 raft_consensus.cc:686] T 6dff6a7601394e41b5ce523f1a243643 P 05272858971e4fa4a3509a4374e60c6c [term 1 LEADER]: Becoming Leader. State: Replica: 05272858971e4fa4a3509a4374e60c6c, State: Running, Role: LEADER
I20260923 17:24:59.299338 17288 consensus_queue.cc:237] T 6dff6a7601394e41b5ce523f1a243643 P 05272858971e4fa4a3509a4374e60c6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca00a23463c43428602e86e93a085c2" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34021 } } peers { permanent_uuid: "05272858971e4fa4a3509a4374e60c6c" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 40839 } } peers { permanent_uuid: "1bd67000226647f38f7dc0ea3e9ebb0e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 37785 } }
I20260923 17:24:59.302817 16814 catalog_manager.cc:5495] T 6dff6a7601394e41b5ce523f1a243643 P 05272858971e4fa4a3509a4374e60c6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 05272858971e4fa4a3509a4374e60c6c (127.11.229.1). New cstate: current_term: 1 leader_uuid: "05272858971e4fa4a3509a4374e60c6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca00a23463c43428602e86e93a085c2" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05272858971e4fa4a3509a4374e60c6c" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 40839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bd67000226647f38f7dc0ea3e9ebb0e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 37785 } health_report { overall_health: UNKNOWN } } }
I20260923 17:24:59.340027 17279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:59.338619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55642 (local address 127.11.229.1:40839)
0923 17:24:59.338752 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:59.338756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:59.338769 (+ 13us) server_negotiation.cc:407] Connection header received
0923 17:24:59.338851 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:59.338853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:59.338915 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:59.338986 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:59.339663 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:59.339667 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:59.339669 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:24:59.339680 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:59.339701 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:24:59.339709 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:59.339788 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:59.339842 (+ 54us) server_negotiation.cc:299] Negotiation successful
0923 17:24:59.339875 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 17:24:59.346226 17088 raft_consensus.cc:1261] T 6dff6a7601394e41b5ce523f1a243643 P 1bd67000226647f38f7dc0ea3e9ebb0e [term 1 FOLLOWER]: Refusing update from remote peer 05272858971e4fa4a3509a4374e60c6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:24:59.346225 17219 raft_consensus.cc:1261] T 6dff6a7601394e41b5ce523f1a243643 P 8ca00a23463c43428602e86e93a085c2 [term 1 FOLLOWER]: Refusing update from remote peer 05272858971e4fa4a3509a4374e60c6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:24:59.346971 17292 consensus_queue.cc:1035] T 6dff6a7601394e41b5ce523f1a243643 P 05272858971e4fa4a3509a4374e60c6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ca00a23463c43428602e86e93a085c2" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:24:59.347155 17288 consensus_queue.cc:1035] T 6dff6a7601394e41b5ce523f1a243643 P 05272858971e4fa4a3509a4374e60c6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bd67000226647f38f7dc0ea3e9ebb0e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 37785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:24:59.354229 17295 mvcc.cc:205] Tried to move back new op lower bound from 7332594890114383872 to 7332594889932926976. Current Snapshot: MvccSnapshot[applied={T|T < 7332594890114383872}]
I20260923 17:24:59.356973 17297 mvcc.cc:205] Tried to move back new op lower bound from 7332594890114383872 to 7332594889932926976. Current Snapshot: MvccSnapshot[applied={T|T < 7332594890114383872}]
I20260923 17:24:59.364890 16813 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:24:59.365051 16813 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:24:59.367952 16813 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6dbd540033d14f1fa67da4fd0325f694: Sending DeleteTablet for 3 replicas of tablet 6dff6a7601394e41b5ce523f1a243643
I20260923 17:24:59.368988 16937 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dff6a7601394e41b5ce523f1a243643 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:24:59 UTC) from {username='slave'} at 127.0.0.1:55634
I20260923 17:24:59.369022 17199 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dff6a7601394e41b5ce523f1a243643 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:24:59 UTC) from {username='slave'} at 127.0.0.1:49470
I20260923 17:24:59.369421 17068 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dff6a7601394e41b5ce523f1a243643 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:24:59 UTC) from {username='slave'} at 127.0.0.1:52676
I20260923 17:24:59.369643 17304 tablet_replica.cc:330] T 6dff6a7601394e41b5ce523f1a243643 P 05272858971e4fa4a3509a4374e60c6c: stopping tablet replica
I20260923 17:24:59.369990 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 16875
I20260923 17:24:59.370038 17304 raft_consensus.cc:2229] T 6dff6a7601394e41b5ce523f1a243643 P 05272858971e4fa4a3509a4374e60c6c [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:24:59.370699 17306 tablet_replica.cc:330] T 6dff6a7601394e41b5ce523f1a243643 P 1bd67000226647f38f7dc0ea3e9ebb0e: stopping tablet replica
I20260923 17:24:59.370955 17306 raft_consensus.cc:2229] T 6dff6a7601394e41b5ce523f1a243643 P 1bd67000226647f38f7dc0ea3e9ebb0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:24:59.371223 17306 raft_consensus.cc:2258] T 6dff6a7601394e41b5ce523f1a243643 P 1bd67000226647f38f7dc0ea3e9ebb0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:24:59.371604 17305 tablet_replica.cc:330] T 6dff6a7601394e41b5ce523f1a243643 P 8ca00a23463c43428602e86e93a085c2: stopping tablet replica
I20260923 17:24:59.371899 17305 raft_consensus.cc:2229] T 6dff6a7601394e41b5ce523f1a243643 P 8ca00a23463c43428602e86e93a085c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:24:59.372179 17305 raft_consensus.cc:2258] T 6dff6a7601394e41b5ce523f1a243643 P 8ca00a23463c43428602e86e93a085c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:24:59.372877 17306 ts_tablet_manager.cc:1905] T 6dff6a7601394e41b5ce523f1a243643 P 1bd67000226647f38f7dc0ea3e9ebb0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:24:59.373471 17305 ts_tablet_manager.cc:1905] T 6dff6a7601394e41b5ce523f1a243643 P 8ca00a23463c43428602e86e93a085c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:24:59.375697 17305 ts_tablet_manager.cc:1918] T 6dff6a7601394e41b5ce523f1a243643 P 8ca00a23463c43428602e86e93a085c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:24:59.375782 17305 log.cc:1196] T 6dff6a7601394e41b5ce523f1a243643 P 8ca00a23463c43428602e86e93a085c2: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-2/wal/wals/6dff6a7601394e41b5ce523f1a243643
I20260923 17:24:59.376009 17305 ts_tablet_manager.cc:1939] T 6dff6a7601394e41b5ce523f1a243643 P 8ca00a23463c43428602e86e93a085c2: Deleting consensus metadata
I20260923 17:24:59.376317 17306 ts_tablet_manager.cc:1918] T 6dff6a7601394e41b5ce523f1a243643 P 1bd67000226647f38f7dc0ea3e9ebb0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:24:59.376411 17306 log.cc:1196] T 6dff6a7601394e41b5ce523f1a243643 P 1bd67000226647f38f7dc0ea3e9ebb0e: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184271597484-12180-0/minicluster-data/ts-1/wal/wals/6dff6a7601394e41b5ce523f1a243643
I20260923 17:24:59.376727 17306 ts_tablet_manager.cc:1939] T 6dff6a7601394e41b5ce523f1a243643 P 1bd67000226647f38f7dc0ea3e9ebb0e: Deleting consensus metadata
I20260923 17:24:59.377400 16802 catalog_manager.cc:4841] TS 8ca00a23463c43428602e86e93a085c2 (127.11.229.3:34021): tablet 6dff6a7601394e41b5ce523f1a243643 (table test-table [id=100e80b78f2a4d879d8b625f859b87fd]) successfully deleted
I20260923 17:24:59.377997 16802 catalog_manager.cc:4841] TS 1bd67000226647f38f7dc0ea3e9ebb0e (127.11.229.2:37785): tablet 6dff6a7601394e41b5ce523f1a243643 (table test-table [id=100e80b78f2a4d879d8b625f859b87fd]) successfully deleted
W20260923 17:24:59.380160 16800 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.229.1:40839 (error 108)
I20260923 17:24:59.380920 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 17006
I20260923 17:24:59.381194 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:59.380891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:40839 (local address 127.0.0.1:55652)
0923 17:24:59.381025 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:59.381098 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.1:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260923 17:24:59.381601 16800 catalog_manager.cc:4584] TS 05272858971e4fa4a3509a4374e60c6c (127.11.229.1:40839): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6dff6a7601394e41b5ce523f1a243643: Network error: Client connection negotiation failed: client connection to 127.11.229.1:40839: connect: Connection refused (error 111)
I20260923 17:24:59.387867 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 17137
I20260923 17:24:59.393756 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 16784
2026-09-23T17:24:59Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1054 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260923 17:24:59.416049 12180 mini_oidc.cc:152] Starting JWKS server
I20260923 17:24:59.417826 12180 webserver.cc:460] Webserver started at https://127.0.0.1:37993/ using document root <none> and password file <none>
I20260923 17:24:59.417966 12180 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 17:24:59.418198 12180 webserver.cc:460] Webserver started at http://127.0.0.1:34979/ using document root <none> and password file <none>
2026-09-23T17:24:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:24:59Z Disabled control of system clock
I20260923 17:24:59.427484 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:46127
--webserver_interface=127.11.229.62
--webserver_port=0
--builtin_ntp_servers=127.11.229.20:35969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:46127
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37993/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184271597484-12180-0/testchainca.cert with env {}
W20260923 17:24:59.539733 17313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:59.540005 17313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:59.540055 17313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:59.544070 17313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:24:59.544124 17313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:59.544140 17313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:59.544157 17313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:24:59.544171 17313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 17:24:59.544188 17313 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 17:24:59.544200 17313 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37993/jwks.json
I20260923 17:24:59.549438 17313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:35969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:46127
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:46127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37993/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184271597484-12180-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:59.550413 17313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:59.551363 17313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:59.559607 17319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:59.559634 17318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:59.559617 17321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:59.559808 17313 server_base.cc:1034] running on GCE node
I20260923 17:24:59.560493 17313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:59.561576 17313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:59.562736 17313 hybrid_clock.cc:648] HybridClock initialized: now 1790184299562709 us; error 40 us; skew 500 ppm
I20260923 17:24:59.564468 17313 webserver.cc:460] Webserver started at http://127.11.229.62:44413/ using document root <none> and password file <none>
I20260923 17:24:59.565037 17313 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:59.565095 17313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:59.565397 17313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:59.567473 17313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "8abee9ac5f23462ea37bdcfd0aba8075"
format_stamp: "Formatted at 2026-09-23 17:24:59 on dist-test-slave-ptl9"
server_key: "404c21e66fd64b461f066819a33d89c1"
server_key_iv: "88ef7ede412eab1ac67000e3f8a46556"
server_key_version: "encryptionkey@0"
I20260923 17:24:59.568079 17313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "8abee9ac5f23462ea37bdcfd0aba8075"
format_stamp: "Formatted at 2026-09-23 17:24:59 on dist-test-slave-ptl9"
server_key: "404c21e66fd64b461f066819a33d89c1"
server_key_iv: "88ef7ede412eab1ac67000e3f8a46556"
server_key_version: "encryptionkey@0"
I20260923 17:24:59.571715 17313 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 17:24:59.574224 17327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:59.575402 17313 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:24:59.575527 17313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "8abee9ac5f23462ea37bdcfd0aba8075"
format_stamp: "Formatted at 2026-09-23 17:24:59 on dist-test-slave-ptl9"
server_key: "404c21e66fd64b461f066819a33d89c1"
server_key_iv: "88ef7ede412eab1ac67000e3f8a46556"
server_key_version: "encryptionkey@0"
I20260923 17:24:59.575634 17313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:59.593756 17313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:59.594463 17313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:59.594676 17313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:59.601951 17313 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:46127
I20260923 17:24:59.601962 17379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:46127 every 8 connection(s)
I20260923 17:24:59.602964 17313 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:24:59.606282 17380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:59.611934 17380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8abee9ac5f23462ea37bdcfd0aba8075: Bootstrap starting.
I20260923 17:24:59.613157 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 17313
I20260923 17:24:59.613325 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:24:59.613610 12180 external_mini_cluster.cc:1428] Setting key 6a660bcc45fc616c352c42338917a3eb
I20260923 17:24:59.614892 17380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8abee9ac5f23462ea37bdcfd0aba8075: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:59.615687 17380 log.cc:824] T 00000000000000000000000000000000 P 8abee9ac5f23462ea37bdcfd0aba8075: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:59.617955 17380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8abee9ac5f23462ea37bdcfd0aba8075: No bootstrap required, opened a new log
I20260923 17:24:59.621809 17380 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8abee9ac5f23462ea37bdcfd0aba8075 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8abee9ac5f23462ea37bdcfd0aba8075" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46127 } }
I20260923 17:24:59.622064 17380 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8abee9ac5f23462ea37bdcfd0aba8075 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:59.622125 17380 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8abee9ac5f23462ea37bdcfd0aba8075 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8abee9ac5f23462ea37bdcfd0aba8075, State: Initialized, Role: FOLLOWER
I20260923 17:24:59.622575 17380 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8abee9ac5f23462ea37bdcfd0aba8075 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8abee9ac5f23462ea37bdcfd0aba8075" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46127 } }
I20260923 17:24:59.622761 17380 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8abee9ac5f23462ea37bdcfd0aba8075 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:24:59.622852 17380 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8abee9ac5f23462ea37bdcfd0aba8075 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:24:59.622987 17380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8abee9ac5f23462ea37bdcfd0aba8075 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:59.623319 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:59.614669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58128 (local address 127.11.229.62:46127)
0923 17:24:59.615396 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:59.615411 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:59.615447 (+ 36us) server_negotiation.cc:407] Connection header received
0923 17:24:59.616277 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:59.616303 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:59.616735 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:59.617089 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:59.617563 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:59.618663 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:59.619583 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:59.620065 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:59.620251 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:59.621157 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:59.621162 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:59.621176 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:24:59.621637 (+ 461us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:59.621693 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:24:59.621714 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:59.621825 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:59.621979 (+ 154us) server_negotiation.cc:299] Negotiation successful
0923 17:24:59.622297 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":140,"threads_started":1}
I20260923 17:24:59.624151 17380 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8abee9ac5f23462ea37bdcfd0aba8075 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8abee9ac5f23462ea37bdcfd0aba8075" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46127 } }
I20260923 17:24:59.624511 17380 leader_election.cc:302] T 00000000000000000000000000000000 P 8abee9ac5f23462ea37bdcfd0aba8075 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8abee9ac5f23462ea37bdcfd0aba8075; no voters:
I20260923 17:24:59.624856 17380 leader_election.cc:288] T 00000000000000000000000000000000 P 8abee9ac5f23462ea37bdcfd0aba8075 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:24:59.624939 17385 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8abee9ac5f23462ea37bdcfd0aba8075 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:24:59.625149 17385 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8abee9ac5f23462ea37bdcfd0aba8075 [term 1 LEADER]: Becoming Leader. State: Replica: 8abee9ac5f23462ea37bdcfd0aba8075, State: Running, Role: LEADER
I20260923 17:24:59.626106 17385 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8abee9ac5f23462ea37bdcfd0aba8075 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8abee9ac5f23462ea37bdcfd0aba8075" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46127 } }
I20260923 17:24:59.626235 17380 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8abee9ac5f23462ea37bdcfd0aba8075 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:24:59.626600 12180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:24:59.627883 17387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8abee9ac5f23462ea37bdcfd0aba8075 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8abee9ac5f23462ea37bdcfd0aba8075. Latest consensus state: current_term: 1 leader_uuid: "8abee9ac5f23462ea37bdcfd0aba8075" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8abee9ac5f23462ea37bdcfd0aba8075" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46127 } } }
I20260923 17:24:59.627915 17386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8abee9ac5f23462ea37bdcfd0aba8075 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8abee9ac5f23462ea37bdcfd0aba8075" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8abee9ac5f23462ea37bdcfd0aba8075" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46127 } } }
I20260923 17:24:59.628003 17387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8abee9ac5f23462ea37bdcfd0aba8075 [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:59.628033 17386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8abee9ac5f23462ea37bdcfd0aba8075 [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:59.628368 17398 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:24:59.631217 17398 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:24:59.636862 17398 catalog_manager.cc:1347] Generated new cluster ID: 4654bbae3ed54c5a8e0c05397980529f
I20260923 17:24:59.636935 17398 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:24:59.650815 17398 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:24:59.651733 17398 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:24:59.659423 17398 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8abee9ac5f23462ea37bdcfd0aba8075: Generated new TSK 0
I20260923 17:24:59.660130 17398 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:24:59.695439 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:59.677959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58144 (local address 127.11.229.62:46127)
0923 17:24:59.678108 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:59.678113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:59.678329 (+ 216us) server_negotiation.cc:407] Connection header received
0923 17:24:59.678547 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:59.678551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:59.678610 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:59.678724 (+ 114us) server_negotiation.cc:226] Negotiated authn=JWT
0923 17:24:59.679003 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:59.679892 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:59.680912 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:59.681295 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:59.681402 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:59.681713 (+ 311us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 17:24:59.694993 (+ 13280us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37993/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 17:24:59.695286 (+ 293us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58144: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37993/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":57}
E20260923 17:24:59.695478 17410 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260923 17:24:59.695572 17410 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260923 17:24:59.695809 17383 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58144: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37993/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260923 17:24:59.696892 17409 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:24:59.677858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46127 (local address 127.0.0.1:58144)
0923 17:24:59.678246 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:59.678259 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:59.678377 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:59.678744 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:59.678746 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:59.678753 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0923 17:24:59.678895 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:59.678901 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:59.679999 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:59.680002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:59.680793 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:59.680803 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:59.681403 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:59.681406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:59.681476 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:59.681519 (+ 43us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 17:24:59.695300 (+ 13781us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37993/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 17:24:59.695602 (+ 302us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.229.62:46127: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37993/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":114,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.11.229.62:46127: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37993/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260923 17:24:59.697814 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 17313
2026-09-23T17:24:59Z chronyd exiting
I20260923 17:24:59.735909 12180 test_util.cc:175] -----------------------------------------------
I20260923 17:24:59.736017 12180 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184271597484-12180-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (321 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260923 17:24:59.737079 12180 mini_oidc.cc:152] Starting JWKS server
I20260923 17:24:59.738019 12180 webserver.cc:460] Webserver started at https://127.0.0.1:32819/ using document root <none> and password file <none>
I20260923 17:24:59.738159 12180 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 17:24:59.738430 12180 webserver.cc:460] Webserver started at http://127.0.0.1:46533/ using document root <none> and password file <none>
2026-09-23T17:24:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:24:59Z Disabled control of system clock
I20260923 17:24:59.747659 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:40849
--webserver_interface=127.11.229.62
--webserver_port=0
--builtin_ntp_servers=127.11.229.20:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:40849
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32819/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184271597484-12180-0/testchainca.cert with env {}
W20260923 17:24:59.858837 17416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:59.859100 17416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:59.859138 17416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:59.863220 17416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:24:59.863274 17416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:59.863289 17416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:59.863306 17416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:24:59.863322 17416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 17:24:59.863379 17416 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 17:24:59.863399 17416 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:32819/jwks.json
I20260923 17:24:59.868654 17416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:40849
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:40849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32819/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184271597484-12180-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:59.869741 17416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:59.870738 17416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:59.879470 17422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:59.879462 17421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:59.879477 17416 server_base.cc:1034] running on GCE node
W20260923 17:24:59.879462 17424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:59.880257 17416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:59.881428 17416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:59.882603 17416 hybrid_clock.cc:648] HybridClock initialized: now 1790184299882585 us; error 28 us; skew 500 ppm
I20260923 17:24:59.884511 17416 webserver.cc:460] Webserver started at http://127.11.229.62:33655/ using document root <none> and password file <none>
I20260923 17:24:59.885123 17416 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:59.885190 17416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:59.885454 17416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:59.887724 17416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "8917a047de464e1c930ff270d7581d8f"
format_stamp: "Formatted at 2026-09-23 17:24:59 on dist-test-slave-ptl9"
server_key: "589a442ccae86e162fd5b8ed38f44762"
server_key_iv: "02e35b5a08357dd289ccfa774f85b5d5"
server_key_version: "encryptionkey@0"
I20260923 17:24:59.888322 17416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "8917a047de464e1c930ff270d7581d8f"
format_stamp: "Formatted at 2026-09-23 17:24:59 on dist-test-slave-ptl9"
server_key: "589a442ccae86e162fd5b8ed38f44762"
server_key_iv: "02e35b5a08357dd289ccfa774f85b5d5"
server_key_version: "encryptionkey@0"
I20260923 17:24:59.892238 17416 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260923 17:24:59.894832 17430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:59.896112 17416 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:24:59.896261 17416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "8917a047de464e1c930ff270d7581d8f"
format_stamp: "Formatted at 2026-09-23 17:24:59 on dist-test-slave-ptl9"
server_key: "589a442ccae86e162fd5b8ed38f44762"
server_key_iv: "02e35b5a08357dd289ccfa774f85b5d5"
server_key_version: "encryptionkey@0"
I20260923 17:24:59.896386 17416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:59.907691 17416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:59.908433 17416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:59.908655 17416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:59.915984 17482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:40849 every 8 connection(s)
I20260923 17:24:59.915982 17416 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:40849
I20260923 17:24:59.917083 17416 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:24:59.920461 17483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:59.923556 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 17416
I20260923 17:24:59.923703 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:24:59.924008 12180 external_mini_cluster.cc:1428] Setting key 72b06e06e0c2443c05ff92c712de6d48
I20260923 17:24:59.926797 17483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8917a047de464e1c930ff270d7581d8f: Bootstrap starting.
I20260923 17:24:59.929052 17483 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8917a047de464e1c930ff270d7581d8f: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:59.929880 17483 log.cc:824] T 00000000000000000000000000000000 P 8917a047de464e1c930ff270d7581d8f: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:59.931874 17483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8917a047de464e1c930ff270d7581d8f: No bootstrap required, opened a new log
I20260923 17:24:59.933595 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:59.925457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38168 (local address 127.11.229.62:40849)
0923 17:24:59.926169 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:59.926185 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:59.926222 (+ 37us) server_negotiation.cc:407] Connection header received
0923 17:24:59.926925 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:59.926956 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:59.927387 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:59.927734 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:59.928186 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:59.929211 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:59.930173 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:59.930666 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:59.930838 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:59.931708 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:59.931714 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:59.931727 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:24:59.932173 (+ 446us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:59.932231 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:24:59.932252 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:59.932365 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:59.932515 (+ 150us) server_negotiation.cc:299] Negotiation successful
0923 17:24:59.932776 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":154,"threads_started":1}
I20260923 17:24:59.936030 17483 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8917a047de464e1c930ff270d7581d8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8917a047de464e1c930ff270d7581d8f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 40849 } }
I20260923 17:24:59.936299 17483 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8917a047de464e1c930ff270d7581d8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:24:59.936381 17483 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8917a047de464e1c930ff270d7581d8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8917a047de464e1c930ff270d7581d8f, State: Initialized, Role: FOLLOWER
I20260923 17:24:59.936571 12180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:24:59.936846 17483 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8917a047de464e1c930ff270d7581d8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8917a047de464e1c930ff270d7581d8f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 40849 } }
I20260923 17:24:59.936978 17483 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8917a047de464e1c930ff270d7581d8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:24:59.937034 17483 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8917a047de464e1c930ff270d7581d8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:24:59.937141 17483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8917a047de464e1c930ff270d7581d8f [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:59.938256 17483 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8917a047de464e1c930ff270d7581d8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8917a047de464e1c930ff270d7581d8f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 40849 } }
I20260923 17:24:59.938609 17483 leader_election.cc:302] T 00000000000000000000000000000000 P 8917a047de464e1c930ff270d7581d8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8917a047de464e1c930ff270d7581d8f; no voters:
I20260923 17:24:59.938928 17483 leader_election.cc:288] T 00000000000000000000000000000000 P 8917a047de464e1c930ff270d7581d8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:24:59.939131 17488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8917a047de464e1c930ff270d7581d8f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:24:59.939359 17488 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8917a047de464e1c930ff270d7581d8f [term 1 LEADER]: Becoming Leader. State: Replica: 8917a047de464e1c930ff270d7581d8f, State: Running, Role: LEADER
I20260923 17:24:59.939671 17488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8917a047de464e1c930ff270d7581d8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8917a047de464e1c930ff270d7581d8f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 40849 } }
I20260923 17:24:59.940189 17483 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8917a047de464e1c930ff270d7581d8f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:24:59.941301 17490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8917a047de464e1c930ff270d7581d8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8917a047de464e1c930ff270d7581d8f. Latest consensus state: current_term: 1 leader_uuid: "8917a047de464e1c930ff270d7581d8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8917a047de464e1c930ff270d7581d8f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 40849 } } }
I20260923 17:24:59.941427 17490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8917a047de464e1c930ff270d7581d8f [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:59.941753 17489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8917a047de464e1c930ff270d7581d8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8917a047de464e1c930ff270d7581d8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8917a047de464e1c930ff270d7581d8f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 40849 } } }
I20260923 17:24:59.941859 17489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8917a047de464e1c930ff270d7581d8f [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:59.941929 17497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:24:59.945405 17497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:24:59.951198 17497 catalog_manager.cc:1347] Generated new cluster ID: 0e00f5f8458044eb9e54aede464cdf92
I20260923 17:24:59.951264 17497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:24:59.966568 17497 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:24:59.967844 17497 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:24:59.975275 17497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8917a047de464e1c930ff270d7581d8f: Generated new TSK 0
I20260923 17:24:59.976125 17497 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260923 17:25:00.005338 17513 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260923 17:25:00.005435 17513 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 17:25:00.006359 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:59.989770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38182 (local address 127.11.229.62:40849)
0923 17:24:59.989923 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:59.989931 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:59.990125 (+ 194us) server_negotiation.cc:407] Connection header received
0923 17:24:59.990291 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:59.990293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:59.990343 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:59.990442 (+ 99us) server_negotiation.cc:226] Negotiated authn=JWT
0923 17:24:59.990708 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:59.991266 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:59.992496 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:59.992716 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:59.992796 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:59.993146 (+ 350us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 17:25:00.005777 (+ 12631us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32819/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 17:25:00.006024 (+ 247us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38182: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32819/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":62}
W20260923 17:25:00.006448 17512 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:24:59.989679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:40849 (local address 127.0.0.1:38182)
0923 17:24:59.990005 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:59.990020 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:59.990156 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:59.990466 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:59.990469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:59.990477 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0923 17:24:59.990602 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:59.990608 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:59.991377 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:59.991380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:59.992376 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:59.992386 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:59.992830 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:59.992834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:59.992943 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:59.992971 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 17:25:00.005990 (+ 13019us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32819/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 17:25:00.006246 (+ 256us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.229.62:40849: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32819/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":119,"threads_started":1}
W20260923 17:25:00.006456 17486 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38182: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32819/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260923 17:25:00.007112 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 17416
2026-09-23T17:25:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (309 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260923 17:25:00.046958 12180 mini_oidc.cc:152] Starting JWKS server
I20260923 17:25:00.047856 12180 webserver.cc:460] Webserver started at https://127.0.0.1:36041/ using document root <none> and password file <none>
I20260923 17:25:00.047971 12180 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 17:25:00.048255 12180 webserver.cc:460] Webserver started at http://127.0.0.1:40041/ using document root <none> and password file <none>
2026-09-23T17:25:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:25:00Z Disabled control of system clock
I20260923 17:25:00.057132 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:46693
--webserver_interface=127.11.229.62
--webserver_port=0
--builtin_ntp_servers=127.11.229.20:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:46693
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36041/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 17:25:00.166038 17519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:00.166332 17519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:00.166401 17519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:00.170320 17519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:25:00.170415 17519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:00.170444 17519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:00.170476 17519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:25:00.170514 17519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 17:25:00.170550 17519 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 17:25:00.170589 17519 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36041/jwks.json
I20260923 17:25:00.175349 17519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:46693
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:46693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36041/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:25:00.176441 17519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:00.177831 17519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:00.186309 17527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:00.186331 17524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:00.186532 17519 server_base.cc:1034] running on GCE node
W20260923 17:25:00.186605 17525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:00.187175 17519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:00.188287 17519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:00.189466 17519 hybrid_clock.cc:648] HybridClock initialized: now 1790184300189448 us; error 31 us; skew 500 ppm
I20260923 17:25:00.191406 17519 webserver.cc:460] Webserver started at http://127.11.229.62:36047/ using document root <none> and password file <none>
I20260923 17:25:00.192183 17519 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:00.192273 17519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:00.192529 17519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:25:00.194849 17519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "ae4294a23d0944f880d46bd1217a5762"
format_stamp: "Formatted at 2026-09-23 17:25:00 on dist-test-slave-ptl9"
server_key: "732e866a291c3ff513527abfcad13d77"
server_key_iv: "2b135113d74493edb851d0601d279477"
server_key_version: "encryptionkey@0"
I20260923 17:25:00.195544 17519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "ae4294a23d0944f880d46bd1217a5762"
format_stamp: "Formatted at 2026-09-23 17:25:00 on dist-test-slave-ptl9"
server_key: "732e866a291c3ff513527abfcad13d77"
server_key_iv: "2b135113d74493edb851d0601d279477"
server_key_version: "encryptionkey@0"
I20260923 17:25:00.199864 17519 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 17:25:00.202543 17533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:00.203967 17519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 17:25:00.204151 17519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "ae4294a23d0944f880d46bd1217a5762"
format_stamp: "Formatted at 2026-09-23 17:25:00 on dist-test-slave-ptl9"
server_key: "732e866a291c3ff513527abfcad13d77"
server_key_iv: "2b135113d74493edb851d0601d279477"
server_key_version: "encryptionkey@0"
I20260923 17:25:00.204277 17519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:00.225972 17519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:00.226851 17519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:00.227062 17519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:00.235244 17519 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:46693
I20260923 17:25:00.235291 17585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:46693 every 8 connection(s)
I20260923 17:25:00.236310 17519 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:25:00.239692 17586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:00.242744 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 17519
I20260923 17:25:00.242869 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:25:00.243160 12180 external_mini_cluster.cc:1428] Setting key 5904ac40033615df39785095e0fb175d
I20260923 17:25:00.246564 17586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae4294a23d0944f880d46bd1217a5762: Bootstrap starting.
I20260923 17:25:00.248962 17586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae4294a23d0944f880d46bd1217a5762: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:00.249835 17586 log.cc:824] T 00000000000000000000000000000000 P ae4294a23d0944f880d46bd1217a5762: Log is configured to *not* fsync() on all Append() calls
I20260923 17:25:00.252358 17586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae4294a23d0944f880d46bd1217a5762: No bootstrap required, opened a new log
I20260923 17:25:00.252931 17589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:00.244634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35056 (local address 127.11.229.62:46693)
0923 17:25:00.245339 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:00.245355 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:00.245392 (+ 37us) server_negotiation.cc:407] Connection header received
0923 17:25:00.246002 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:00.246020 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:00.246415 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:00.246768 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:00.247219 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.248243 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.249304 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.249801 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.249993 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:00.250884 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:00.250891 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:00.250911 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:25:00.251359 (+ 448us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:00.251418 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:25:00.251439 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:00.251519 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:00.251740 (+ 221us) server_negotiation.cc:299] Negotiation successful
0923 17:25:00.252059 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":133,"threads_started":1}
I20260923 17:25:00.256094 12180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:25:00.256976 17586 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae4294a23d0944f880d46bd1217a5762 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4294a23d0944f880d46bd1217a5762" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46693 } }
I20260923 17:25:00.257382 17586 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae4294a23d0944f880d46bd1217a5762 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:00.257473 17586 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae4294a23d0944f880d46bd1217a5762 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4294a23d0944f880d46bd1217a5762, State: Initialized, Role: FOLLOWER
I20260923 17:25:00.257997 17586 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae4294a23d0944f880d46bd1217a5762 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4294a23d0944f880d46bd1217a5762" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46693 } }
I20260923 17:25:00.258163 17586 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae4294a23d0944f880d46bd1217a5762 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:25:00.258266 17586 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae4294a23d0944f880d46bd1217a5762 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:25:00.258368 17586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae4294a23d0944f880d46bd1217a5762 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:00.307826 17586 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae4294a23d0944f880d46bd1217a5762 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4294a23d0944f880d46bd1217a5762" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46693 } }
I20260923 17:25:00.308434 17586 leader_election.cc:302] T 00000000000000000000000000000000 P ae4294a23d0944f880d46bd1217a5762 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae4294a23d0944f880d46bd1217a5762; no voters:
I20260923 17:25:00.308887 17586 leader_election.cc:288] T 00000000000000000000000000000000 P ae4294a23d0944f880d46bd1217a5762 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:25:00.309042 17591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae4294a23d0944f880d46bd1217a5762 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:25:00.309346 17591 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae4294a23d0944f880d46bd1217a5762 [term 1 LEADER]: Becoming Leader. State: Replica: ae4294a23d0944f880d46bd1217a5762, State: Running, Role: LEADER
I20260923 17:25:00.309736 17591 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae4294a23d0944f880d46bd1217a5762 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4294a23d0944f880d46bd1217a5762" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46693 } }
I20260923 17:25:00.310431 17586 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae4294a23d0944f880d46bd1217a5762 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:25:00.311815 17593 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae4294a23d0944f880d46bd1217a5762 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae4294a23d0944f880d46bd1217a5762. Latest consensus state: current_term: 1 leader_uuid: "ae4294a23d0944f880d46bd1217a5762" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4294a23d0944f880d46bd1217a5762" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46693 } } }
I20260923 17:25:00.311954 17593 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae4294a23d0944f880d46bd1217a5762 [sys.catalog]: This master's current role is: LEADER
I20260923 17:25:00.312479 17600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:25:00.312510 17592 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae4294a23d0944f880d46bd1217a5762 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae4294a23d0944f880d46bd1217a5762" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4294a23d0944f880d46bd1217a5762" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46693 } } }
I20260923 17:25:00.312634 17592 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae4294a23d0944f880d46bd1217a5762 [sys.catalog]: This master's current role is: LEADER
I20260923 17:25:00.315678 17600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:25:00.321244 17600 catalog_manager.cc:1347] Generated new cluster ID: 2e6b69c7763d4d0da57a36f08b5fbd49
I20260923 17:25:00.321383 17600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:25:00.340628 17600 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:25:00.342034 17600 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:25:00.351805 17600 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae4294a23d0944f880d46bd1217a5762: Generated new TSK 0
I20260923 17:25:00.352730 17600 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:25:00.376329 17589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:00.370802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35058 (local address 127.11.229.62:46693)
0923 17:25:00.370944 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:00.370949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:00.371112 (+ 163us) server_negotiation.cc:407] Connection header received
0923 17:25:00.371265 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:00.371268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:00.371333 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:00.371462 (+ 129us) server_negotiation.cc:226] Negotiated authn=JWT
0923 17:25:00.371673 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.372520 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.373476 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.373863 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.373970 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:00.374293 (+ 323us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 17:25:00.375925 (+ 1632us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36041/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 17:25:00.376176 (+ 251us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35058: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36041/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":58}
W20260923 17:25:00.376410 17589 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35058: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36041/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260923 17:25:00.377998 17615 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:00.370716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46693 (local address 127.0.0.1:35058)
0923 17:25:00.371010 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:00.371024 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:00.371140 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:00.371459 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:00.371461 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:00.371468 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0923 17:25:00.371571 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:00.371575 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.372642 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.372645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:00.373361 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:00.373371 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.373987 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.373990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:00.374065 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:00.374089 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 17:25:00.376095 (+ 2006us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36041/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 17:25:00.377823 (+ 1728us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.229.62:46693: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36041/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":1508,"thread_start_us":126,"threads_started":1}
I20260923 17:25:00.378612 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 17519
E20260923 17:25:00.387102 17616 webserver.cc:533] Webserver: sslize failed:
E20260923 17:25:00.387185 17616 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-23T17:25:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (371 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-23T17:25:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:25:00Z Disabled control of system clock
I20260923 17:25:00.427224 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:40447
--webserver_interface=127.11.229.62
--webserver_port=0
--builtin_ntp_servers=127.11.229.20:33587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:40447
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184271597484-12180-0/testchainca.cert with env {}
W20260923 17:25:00.543262 17620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:00.543524 17620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:00.543561 17620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:00.547468 17620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:25:00.547518 17620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:00.547533 17620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:00.547550 17620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:25:00.547565 17620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 17:25:00.547581 17620 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 17:25:00.547595 17620 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260923 17:25:00.552896 17620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:33587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:40447
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:40447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184271597484-12180-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:25:00.553979 17620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:00.554922 17620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:00.564167 17628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:00.564214 17620 server_base.cc:1034] running on GCE node
W20260923 17:25:00.564154 17626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:00.564153 17625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:00.565079 17620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:00.566279 17620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:00.567440 17620 hybrid_clock.cc:648] HybridClock initialized: now 1790184300567406 us; error 48 us; skew 500 ppm
I20260923 17:25:00.569478 17620 webserver.cc:460] Webserver started at http://127.11.229.62:44845/ using document root <none> and password file <none>
I20260923 17:25:00.570068 17620 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:00.570132 17620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:00.570369 17620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:25:00.572454 17620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "eadea1d1fbc44dc782543a1aebb11c46"
format_stamp: "Formatted at 2026-09-23 17:25:00 on dist-test-slave-ptl9"
server_key: "fe0eaf546102727bf80ec05300adba3e"
server_key_iv: "d87109f418e4198e23d7a3ccb23d9500"
server_key_version: "encryptionkey@0"
I20260923 17:25:00.573031 17620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "eadea1d1fbc44dc782543a1aebb11c46"
format_stamp: "Formatted at 2026-09-23 17:25:00 on dist-test-slave-ptl9"
server_key: "fe0eaf546102727bf80ec05300adba3e"
server_key_iv: "d87109f418e4198e23d7a3ccb23d9500"
server_key_version: "encryptionkey@0"
I20260923 17:25:00.576673 17620 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:25:00.579138 17634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:00.580345 17620 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:25:00.580485 17620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "eadea1d1fbc44dc782543a1aebb11c46"
format_stamp: "Formatted at 2026-09-23 17:25:00 on dist-test-slave-ptl9"
server_key: "fe0eaf546102727bf80ec05300adba3e"
server_key_iv: "d87109f418e4198e23d7a3ccb23d9500"
server_key_version: "encryptionkey@0"
I20260923 17:25:00.580588 17620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:00.597469 17620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:00.598192 17620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:00.598387 17620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:00.605865 17620 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:40447
I20260923 17:25:00.605890 17686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:40447 every 8 connection(s)
I20260923 17:25:00.606827 17620 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:25:00.609699 17687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:00.613073 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 17620
I20260923 17:25:00.613237 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:25:00.613550 12180 external_mini_cluster.cc:1428] Setting key d424857e4b285851d224ea792a879014
I20260923 17:25:00.616031 17687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eadea1d1fbc44dc782543a1aebb11c46: Bootstrap starting.
I20260923 17:25:00.618429 17687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eadea1d1fbc44dc782543a1aebb11c46: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:00.619201 17687 log.cc:824] T 00000000000000000000000000000000 P eadea1d1fbc44dc782543a1aebb11c46: Log is configured to *not* fsync() on all Append() calls
I20260923 17:25:00.622102 17687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eadea1d1fbc44dc782543a1aebb11c46: No bootstrap required, opened a new log
I20260923 17:25:00.623386 17690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:00.614969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36376 (local address 127.11.229.62:40447)
0923 17:25:00.615720 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:00.615736 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:00.615774 (+ 38us) server_negotiation.cc:407] Connection header received
0923 17:25:00.616504 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:00.616524 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:00.616950 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:00.617337 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:00.617812 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.618911 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.620032 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.620512 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.620653 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:00.621554 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:00.621560 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:00.621574 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:25:00.621981 (+ 407us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:00.622040 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:25:00.622061 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:00.622178 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:00.622352 (+ 174us) server_negotiation.cc:299] Negotiation successful
0923 17:25:00.622670 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":148,"threads_started":1}
I20260923 17:25:00.626493 12180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:25:00.626708 17687 raft_consensus.cc:348] T 00000000000000000000000000000000 P eadea1d1fbc44dc782543a1aebb11c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadea1d1fbc44dc782543a1aebb11c46" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 40447 } }
I20260923 17:25:00.626960 17687 raft_consensus.cc:374] T 00000000000000000000000000000000 P eadea1d1fbc44dc782543a1aebb11c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:00.627036 17687 raft_consensus.cc:729] T 00000000000000000000000000000000 P eadea1d1fbc44dc782543a1aebb11c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eadea1d1fbc44dc782543a1aebb11c46, State: Initialized, Role: FOLLOWER
I20260923 17:25:00.627525 17687 consensus_queue.cc:260] T 00000000000000000000000000000000 P eadea1d1fbc44dc782543a1aebb11c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadea1d1fbc44dc782543a1aebb11c46" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 40447 } }
I20260923 17:25:00.627698 17687 raft_consensus.cc:388] T 00000000000000000000000000000000 P eadea1d1fbc44dc782543a1aebb11c46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:25:00.627786 17687 raft_consensus.cc:482] T 00000000000000000000000000000000 P eadea1d1fbc44dc782543a1aebb11c46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:25:00.627887 17687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eadea1d1fbc44dc782543a1aebb11c46 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:00.628937 17687 raft_consensus.cc:504] T 00000000000000000000000000000000 P eadea1d1fbc44dc782543a1aebb11c46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadea1d1fbc44dc782543a1aebb11c46" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 40447 } }
I20260923 17:25:00.629338 17687 leader_election.cc:302] T 00000000000000000000000000000000 P eadea1d1fbc44dc782543a1aebb11c46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eadea1d1fbc44dc782543a1aebb11c46; no voters:
I20260923 17:25:00.629663 17687 leader_election.cc:288] T 00000000000000000000000000000000 P eadea1d1fbc44dc782543a1aebb11c46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:25:00.629741 17692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eadea1d1fbc44dc782543a1aebb11c46 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:25:00.630026 17692 raft_consensus.cc:686] T 00000000000000000000000000000000 P eadea1d1fbc44dc782543a1aebb11c46 [term 1 LEADER]: Becoming Leader. State: Replica: eadea1d1fbc44dc782543a1aebb11c46, State: Running, Role: LEADER
I20260923 17:25:00.630374 17692 consensus_queue.cc:237] T 00000000000000000000000000000000 P eadea1d1fbc44dc782543a1aebb11c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadea1d1fbc44dc782543a1aebb11c46" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 40447 } }
I20260923 17:25:00.630906 17687 sys_catalog.cc:564] T 00000000000000000000000000000000 P eadea1d1fbc44dc782543a1aebb11c46 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:25:00.632048 17694 sys_catalog.cc:455] T 00000000000000000000000000000000 P eadea1d1fbc44dc782543a1aebb11c46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eadea1d1fbc44dc782543a1aebb11c46. Latest consensus state: current_term: 1 leader_uuid: "eadea1d1fbc44dc782543a1aebb11c46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadea1d1fbc44dc782543a1aebb11c46" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 40447 } } }
I20260923 17:25:00.632184 17694 sys_catalog.cc:458] T 00000000000000000000000000000000 P eadea1d1fbc44dc782543a1aebb11c46 [sys.catalog]: This master's current role is: LEADER
I20260923 17:25:00.632572 17701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:25:00.633579 17693 sys_catalog.cc:455] T 00000000000000000000000000000000 P eadea1d1fbc44dc782543a1aebb11c46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eadea1d1fbc44dc782543a1aebb11c46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadea1d1fbc44dc782543a1aebb11c46" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 40447 } } }
I20260923 17:25:00.633695 17693 sys_catalog.cc:458] T 00000000000000000000000000000000 P eadea1d1fbc44dc782543a1aebb11c46 [sys.catalog]: This master's current role is: LEADER
I20260923 17:25:00.635937 17701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:25:00.641670 17701 catalog_manager.cc:1347] Generated new cluster ID: d3a2d92e0755489e96a41c9d454e7264
I20260923 17:25:00.641760 17701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:25:00.653434 17701 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:25:00.654405 17701 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:25:00.661309 17701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eadea1d1fbc44dc782543a1aebb11c46: Generated new TSK 0
I20260923 17:25:00.662147 17701 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:25:00.730249 17690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:00.678185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36386 (local address 127.11.229.62:40447)
0923 17:25:00.678343 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:00.678348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:00.678560 (+ 212us) server_negotiation.cc:407] Connection header received
0923 17:25:00.678729 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:00.678731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:00.678779 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:00.678870 (+ 91us) server_negotiation.cc:226] Negotiated authn=JWT
0923 17:25:00.679133 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.679682 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.680906 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.681121 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.681196 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:00.681567 (+ 371us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 17:25:00.729856 (+ 48289us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 17:25:00.730130 (+ 274us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36386: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":66}
W20260923 17:25:00.730360 17690 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36386: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260923 17:25:00.730532 17716 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:00.678137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:40447 (local address 127.0.0.1:36386)
0923 17:25:00.678463 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:00.678479 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:00.678596 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:00.678893 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:00.678896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:00.678904 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0923 17:25:00.679025 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:00.679031 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.679795 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.679799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:00.680785 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:00.680796 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:00.681241 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:00.681245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:00.681363 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:00.681393 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 17:25:00.730115 (+ 48722us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 17:25:00.730340 (+ 225us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.229.62:40447: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":134,"threads_started":1}
I20260923 17:25:00.731148 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 17620
2026-09-23T17:25:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (332 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (125 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (125 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:25:01 dist-test-slave-ptl9 krb5kdc[17728](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:25:01 dist-test-slave-ptl9 krb5kdc[17728](info): set up 2 sockets
Sep 23 17:25:01 dist-test-slave-ptl9 krb5kdc[17728](info): commencing operation
krb5kdc: starting...
W20260923 17:25:02.017889 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:25:02 dist-test-slave-ptl9 krb5kdc[17728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:25:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:25:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.229.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
I20260923 17:25:02.140443 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:42357
--webserver_interface=127.11.229.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:45919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:42357
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:25:02.251610 17741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:02.251897 17741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:02.251937 17741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:02.255998 17741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:25:02.256062 17741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:02.256080 17741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:02.256098 17741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:25:02.256112 17741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:25:02.261487 17741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:45919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:42357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:42357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:25:02.262574 17741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:02.263582 17741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:02.272545 17746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:02.272562 17741 server_base.cc:1034] running on GCE node
W20260923 17:25:02.272498 17747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:02.272485 17749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:02.273411 17741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:02.274573 17741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:02.275777 17741 hybrid_clock.cc:648] HybridClock initialized: now 1790184302275745 us; error 50 us; skew 500 ppm
Sep 23 17:25:02 dist-test-slave-ptl9 krb5kdc[17728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184302, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:02.278580 17741 init.cc:377] Logged in from keytab as kudu/127.11.229.62@KRBTEST.COM (short username kudu)
I20260923 17:25:02.279577 17741 webserver.cc:460] Webserver started at http://127.11.229.62:41397/ using document root <none> and password file <none>
I20260923 17:25:02.280185 17741 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:02.280279 17741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:02.280526 17741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:25:02.282761 17741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "5bbd981a66a8426e943fc8bc890a1323"
format_stamp: "Formatted at 2026-09-23 17:25:02 on dist-test-slave-ptl9"
server_key: "83dd0c8e6dc2f9ebc16d9a926e3edb67"
server_key_iv: "5a9bacee6fd6ff240ad7a549bfa11a5b"
server_key_version: "encryptionkey@0"
I20260923 17:25:02.283355 17741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "5bbd981a66a8426e943fc8bc890a1323"
format_stamp: "Formatted at 2026-09-23 17:25:02 on dist-test-slave-ptl9"
server_key: "83dd0c8e6dc2f9ebc16d9a926e3edb67"
server_key_iv: "5a9bacee6fd6ff240ad7a549bfa11a5b"
server_key_version: "encryptionkey@0"
I20260923 17:25:02.287343 17741 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:25:02.289924 17756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:02.291225 17741 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:25:02.291384 17741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "5bbd981a66a8426e943fc8bc890a1323"
format_stamp: "Formatted at 2026-09-23 17:25:02 on dist-test-slave-ptl9"
server_key: "83dd0c8e6dc2f9ebc16d9a926e3edb67"
server_key_iv: "5a9bacee6fd6ff240ad7a549bfa11a5b"
server_key_version: "encryptionkey@0"
I20260923 17:25:02.291507 17741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:02.311635 17741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:02.314766 17741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:02.314981 17741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:02.322376 17741 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:42357
I20260923 17:25:02.322392 17808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:42357 every 8 connection(s)
I20260923 17:25:02.323405 17741 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:25:02.326442 17809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:02.327257 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 17741
I20260923 17:25:02.327390 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:25:02.327715 12180 external_mini_cluster.cc:1428] Setting key a9f726a447e8d3c1eb47b0b84414f14d
I20260923 17:25:02.333381 17809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bbd981a66a8426e943fc8bc890a1323: Bootstrap starting.
I20260923 17:25:02.336299 17809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bbd981a66a8426e943fc8bc890a1323: Neither blocks nor log segments found. Creating new log.
Sep 23 17:25:02 dist-test-slave-ptl9 krb5kdc[17728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:25:02.337149 17809 log.cc:824] T 00000000000000000000000000000000 P 5bbd981a66a8426e943fc8bc890a1323: Log is configured to *not* fsync() on all Append() calls
I20260923 17:25:02.339215 17809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bbd981a66a8426e943fc8bc890a1323: No bootstrap required, opened a new log
I20260923 17:25:02.342226 17812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:02.329148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37168 (local address 127.11.229.62:42357)
0923 17:25:02.329624 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:02.329644 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:02.329681 (+ 37us) server_negotiation.cc:407] Connection header received
0923 17:25:02.330347 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:02.330365 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:02.330714 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:02.331001 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:02.331948 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:02.332955 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:02.334099 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:02.334567 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:02.334685 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:02.337594 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:02.337600 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:02.337615 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:02.337649 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:02.339353 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:02.339809 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:02.339823 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:02.339828 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:02.339938 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:02.340226 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:02.340230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:02.340232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:02.340646 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:02.340848 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:02.341143 (+ 295us) server_negotiation.cc:299] Negotiation successful
0923 17:25:02.341482 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":189,"threads_started":1}
I20260923 17:25:02.343019 17809 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bbd981a66a8426e943fc8bc890a1323 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bbd981a66a8426e943fc8bc890a1323" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 42357 } }
I20260923 17:25:02.343273 17809 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bbd981a66a8426e943fc8bc890a1323 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:02.343334 17809 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bbd981a66a8426e943fc8bc890a1323 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bbd981a66a8426e943fc8bc890a1323, State: Initialized, Role: FOLLOWER
I20260923 17:25:02.343855 17809 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bbd981a66a8426e943fc8bc890a1323 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bbd981a66a8426e943fc8bc890a1323" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 42357 } }
I20260923 17:25:02.343992 17809 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bbd981a66a8426e943fc8bc890a1323 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:25:02.344079 17809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bbd981a66a8426e943fc8bc890a1323 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:25:02.344218 17809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bbd981a66a8426e943fc8bc890a1323 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:02.345398 17809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bbd981a66a8426e943fc8bc890a1323 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bbd981a66a8426e943fc8bc890a1323" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 42357 } }
I20260923 17:25:02.345747 17809 leader_election.cc:302] T 00000000000000000000000000000000 P 5bbd981a66a8426e943fc8bc890a1323 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bbd981a66a8426e943fc8bc890a1323; no voters:
I20260923 17:25:02.346065 17809 leader_election.cc:288] T 00000000000000000000000000000000 P 5bbd981a66a8426e943fc8bc890a1323 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:25:02.346210 17814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bbd981a66a8426e943fc8bc890a1323 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:25:02.346469 17814 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bbd981a66a8426e943fc8bc890a1323 [term 1 LEADER]: Becoming Leader. State: Replica: 5bbd981a66a8426e943fc8bc890a1323, State: Running, Role: LEADER
I20260923 17:25:02.346774 17814 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bbd981a66a8426e943fc8bc890a1323 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bbd981a66a8426e943fc8bc890a1323" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 42357 } }
I20260923 17:25:02.347324 17809 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bbd981a66a8426e943fc8bc890a1323 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:25:02.348459 17815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bbd981a66a8426e943fc8bc890a1323 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bbd981a66a8426e943fc8bc890a1323" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bbd981a66a8426e943fc8bc890a1323" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 42357 } } }
I20260923 17:25:02.348598 17815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bbd981a66a8426e943fc8bc890a1323 [sys.catalog]: This master's current role is: LEADER
I20260923 17:25:02.348470 17816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bbd981a66a8426e943fc8bc890a1323 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bbd981a66a8426e943fc8bc890a1323. Latest consensus state: current_term: 1 leader_uuid: "5bbd981a66a8426e943fc8bc890a1323" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bbd981a66a8426e943fc8bc890a1323" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 42357 } } }
I20260923 17:25:02.348682 17816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bbd981a66a8426e943fc8bc890a1323 [sys.catalog]: This master's current role is: LEADER
W20260923 17:25:02.352792 17830 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5bbd981a66a8426e943fc8bc890a1323: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:25:02.352880 17830 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:25:02.353044 17823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:25:02.354120 17823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:25:02.359839 17823 catalog_manager.cc:1347] Generated new cluster ID: 3a7b0b6877ff441089f42624c07c6def
I20260923 17:25:02.359944 17823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:25:02.385767 17823 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:25:02.387120 17823 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:25:02.393666 17823 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bbd981a66a8426e943fc8bc890a1323: Generated new TSK 0
I20260923 17:25:02.394487 17823 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.229.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab.
Entry for principal kudu/127.11.229.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab.
I20260923 17:25:02.433673 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.1:0
--local_ip_for_outbound_sockets=127.11.229.1
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:42357
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:45919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:25:02.555909 17835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:02.556249 17835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:02.556296 17835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:02.560698 17835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:02.560770 17835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:02.560832 17835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.1
I20260923 17:25:02.566547 17835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:45919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:42357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.1
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:25:02.567857 17835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:02.568938 17835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:02.579059 17843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:02.579062 17841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:02.579104 17840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:02.579305 17835 server_base.cc:1034] running on GCE node
I20260923 17:25:02.579738 17835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:02.580423 17835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:02.581611 17835 hybrid_clock.cc:648] HybridClock initialized: now 1790184302581595 us; error 28 us; skew 500 ppm
Sep 23 17:25:02 dist-test-slave-ptl9 krb5kdc[17728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184302, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:02.584453 17835 init.cc:377] Logged in from keytab as kudu/127.11.229.1@KRBTEST.COM (short username kudu)
I20260923 17:25:02.585400 17835 webserver.cc:460] Webserver started at http://127.11.229.1:46057/ using document root <none> and password file <none>
I20260923 17:25:02.585983 17835 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:02.586071 17835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:02.586320 17835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:25:02.588444 17835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-0/data/instance:
uuid: "71f799af2bc240a98205ae8ef7a4dd98"
format_stamp: "Formatted at 2026-09-23 17:25:02 on dist-test-slave-ptl9"
server_key: "2b4c862f9dd491581c207583bc7d43bd"
server_key_iv: "532b942d1fcec2a4d25718a30847f881"
server_key_version: "encryptionkey@0"
I20260923 17:25:02.589017 17835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance:
uuid: "71f799af2bc240a98205ae8ef7a4dd98"
format_stamp: "Formatted at 2026-09-23 17:25:02 on dist-test-slave-ptl9"
server_key: "2b4c862f9dd491581c207583bc7d43bd"
server_key_iv: "532b942d1fcec2a4d25718a30847f881"
server_key_version: "encryptionkey@0"
I20260923 17:25:02.592947 17835 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260923 17:25:02.595646 17850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:02.596989 17835 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:25:02.597242 17835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-0/wal
uuid: "71f799af2bc240a98205ae8ef7a4dd98"
format_stamp: "Formatted at 2026-09-23 17:25:02 on dist-test-slave-ptl9"
server_key: "2b4c862f9dd491581c207583bc7d43bd"
server_key_iv: "532b942d1fcec2a4d25718a30847f881"
server_key_version: "encryptionkey@0"
I20260923 17:25:02.597422 17835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:02.610119 17835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:02.613787 17835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:02.614015 17835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:02.614650 17835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:25:02.615542 17835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:25:02.615614 17835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:02.615690 17835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:25:02.615734 17835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:02.624538 17835 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.1:44819
I20260923 17:25:02.624627 17963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.1:44819 every 8 connection(s)
I20260923 17:25:02.625589 17835 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
I20260923 17:25:02.630507 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 17835
I20260923 17:25:02.630662 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance
I20260923 17:25:02.630942 12180 external_mini_cluster.cc:1428] Setting key 0166ac05b7febb72360a5fa996576997
Sep 23 17:25:02 dist-test-slave-ptl9 krb5kdc[17728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184302, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.229.2@KRBTEST.COM; defaulting to no policy
I20260923 17:25:02.641089 17812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:02.627544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:44669 (local address 127.11.229.62:42357)
0923 17:25:02.627694 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:02.627698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:02.628485 (+ 787us) server_negotiation.cc:407] Connection header received
0923 17:25:02.629996 (+ 1511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:02.629999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:02.630065 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:02.630166 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:02.631533 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:02.632266 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:02.633572 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:02.633855 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:02.634005 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:02.636688 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:02.636694 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:02.636699 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:02.636732 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:02.638525 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:02.639237 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:02.639239 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:02.639241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:02.639328 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:02.639739 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:02.639742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:02.639743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:02.639928 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:02.640011 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:02.640777 (+ 766us) server_negotiation.cc:299] Negotiation successful
0923 17:25:02.640911 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 17:25:02.642117 17966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:02.627873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:42357 (local address 127.11.229.1:44669)
0923 17:25:02.628329 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:02.628374 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:02.629761 (+ 1387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:02.630372 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:02.630379 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:02.630834 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:02.631380 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:02.631394 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:02.632404 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:02.632408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:02.633414 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:02.633425 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:02.634067 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:02.634071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:02.634247 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:02.634887 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:02.634904 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:02.636494 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:02.638666 (+ 2172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:02.638673 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:02.638694 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:02.639113 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:02.639429 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:02.639432 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:02.639437 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:02.639638 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:02.640056 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:02.640065 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:02.640455 (+ 390us) client_negotiation.cc:769] Sending connection context
0923 17:25:02.640693 (+ 238us) client_negotiation.cc:240] Negotiation successful
0923 17:25:02.641110 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":119,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.2@KRBTEST.COM" created.
I20260923 17:25:02.643525 17964 heartbeater.cc:347] Connected to a master server at 127.11.229.62:42357
I20260923 17:25:02.643782 17964 heartbeater.cc:464] Registering TS with master...
I20260923 17:25:02.644361 17964 heartbeater.cc:511] Master 127.11.229.62:42357 requested a full tablet report, sending...
I20260923 17:25:02.645989 17773 ts_manager.cc:194] Registered new tserver with Master: 71f799af2bc240a98205ae8ef7a4dd98 (127.11.229.1:44819)
I20260923 17:25:02.647272 17773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.1@KRBTEST.COM'} at 127.11.229.1:44669
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab.
Entry for principal kudu/127.11.229.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab.
I20260923 17:25:02.660112 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.2:0
--local_ip_for_outbound_sockets=127.11.229.2
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:42357
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:45919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:25:02.774838 17969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:02.775192 17969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:02.775269 17969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:02.779506 17969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:02.779620 17969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:02.779716 17969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.2
I20260923 17:25:02.785427 17969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:45919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:42357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.2
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:25:02.786716 17969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:02.787796 17969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:02.798195 17975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:02.798213 17974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:02.798236 17977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:02.798569 17969 server_base.cc:1034] running on GCE node
I20260923 17:25:02.799026 17969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:02.799713 17969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:02.800894 17969 hybrid_clock.cc:648] HybridClock initialized: now 1790184302800861 us; error 41 us; skew 500 ppm
Sep 23 17:25:02 dist-test-slave-ptl9 krb5kdc[17728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184302, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:02.803906 17969 init.cc:377] Logged in from keytab as kudu/127.11.229.2@KRBTEST.COM (short username kudu)
I20260923 17:25:02.804988 17969 webserver.cc:460] Webserver started at http://127.11.229.2:43195/ using document root <none> and password file <none>
I20260923 17:25:02.805675 17969 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:02.805771 17969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:02.806012 17969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:25:02.808307 17969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-1/data/instance:
uuid: "3ee218478eef4d7794d02317089c4657"
format_stamp: "Formatted at 2026-09-23 17:25:02 on dist-test-slave-ptl9"
server_key: "74e17aab2471615a584946f2c4e58eef"
server_key_iv: "376e61368ffb4a98ec02d4040016901e"
server_key_version: "encryptionkey@0"
I20260923 17:25:02.808923 17969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance:
uuid: "3ee218478eef4d7794d02317089c4657"
format_stamp: "Formatted at 2026-09-23 17:25:02 on dist-test-slave-ptl9"
server_key: "74e17aab2471615a584946f2c4e58eef"
server_key_iv: "376e61368ffb4a98ec02d4040016901e"
server_key_version: "encryptionkey@0"
I20260923 17:25:02.812932 17969 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 17:25:02.815603 17984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:02.816912 17969 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260923 17:25:02.817052 17969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-1/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-1/wal
uuid: "3ee218478eef4d7794d02317089c4657"
format_stamp: "Formatted at 2026-09-23 17:25:02 on dist-test-slave-ptl9"
server_key: "74e17aab2471615a584946f2c4e58eef"
server_key_iv: "376e61368ffb4a98ec02d4040016901e"
server_key_version: "encryptionkey@0"
I20260923 17:25:02.817188 17969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:02.833813 17969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:02.837209 17969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:02.837458 17969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:02.838193 17969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:25:02.839107 17969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:25:02.839179 17969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:02.839251 17969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:25:02.839305 17969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:02.848747 17969 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.2:33583
I20260923 17:25:02.848778 18097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.2:33583 every 8 connection(s)
I20260923 17:25:02.849843 17969 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
I20260923 17:25:02.857303 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 17969
I20260923 17:25:02.857446 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance
I20260923 17:25:02.857738 12180 external_mini_cluster.cc:1428] Setting key 5ecb50810e5b4b7072636cd8eecfa4c5
Sep 23 17:25:02 dist-test-slave-ptl9 krb5kdc[17728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184302, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:25:02.865149 17812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:02.851798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:35719 (local address 127.11.229.62:42357)
0923 17:25:02.851955 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:02.851959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:02.852826 (+ 867us) server_negotiation.cc:407] Connection header received
0923 17:25:02.854340 (+ 1514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:02.854343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:02.854403 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:02.854508 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:02.855733 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:02.856287 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:02.857969 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:02.858202 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:02.858303 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:02.861401 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:02.861405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:02.861408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:02.861444 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:02.862932 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:02.863514 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:02.863518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:02.863521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:02.863613 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:02.863979 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:02.863982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:02.863984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:02.864185 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:02.864297 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:02.864850 (+ 553us) server_negotiation.cc:299] Negotiation successful
0923 17:25:02.864980 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:25:02.866124 18100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:02.852206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:42357 (local address 127.11.229.2:35719)
0923 17:25:02.852665 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:02.852711 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:02.854106 (+ 1395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:02.854702 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:02.854710 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:02.855073 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:02.855573 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:02.855587 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:02.856509 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:02.856512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:02.857818 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:02.857834 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:02.858387 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:02.858392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:02.858601 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:02.859344 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:02.859368 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:02.861176 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:02.863069 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:02.863076 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:02.863102 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:02.863393 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:02.863707 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:02.863712 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:02.863714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:02.863871 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:02.864295 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:02.864303 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:02.864583 (+ 280us) client_negotiation.cc:769] Sending connection context
0923 17:25:02.864790 (+ 207us) client_negotiation.cc:240] Negotiation successful
0923 17:25:02.865165 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":116,"threads_started":1}
WARNING: no policy specified for kudu/127.11.229.3@KRBTEST.COM; defaulting to no policy
I20260923 17:25:02.868203 18098 heartbeater.cc:347] Connected to a master server at 127.11.229.62:42357
I20260923 17:25:02.868535 18098 heartbeater.cc:464] Registering TS with master...
I20260923 17:25:02.869102 18098 heartbeater.cc:511] Master 127.11.229.62:42357 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.3@KRBTEST.COM" created.
I20260923 17:25:02.870481 17773 ts_manager.cc:194] Registered new tserver with Master: 3ee218478eef4d7794d02317089c4657 (127.11.229.2:33583)
I20260923 17:25:02.871356 17773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.2@KRBTEST.COM'} at 127.11.229.2:35719
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab.
Entry for principal kudu/127.11.229.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab.
I20260923 17:25:02.886615 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.3:0
--local_ip_for_outbound_sockets=127.11.229.3
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:42357
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:45919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:25:03.002575 18103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:03.002859 18103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:03.002903 18103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:03.007355 18103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:03.007422 18103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:03.007484 18103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.3
I20260923 17:25:03.013124 18103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:45919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:42357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.3
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:25:03.014325 18103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:03.015398 18103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:03.025664 18111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:03.025686 18108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:03.025732 18103 server_base.cc:1034] running on GCE node
W20260923 17:25:03.025687 18109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:03.026311 18103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:03.027020 18103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:03.028242 18103 hybrid_clock.cc:648] HybridClock initialized: now 1790184303028203 us; error 42 us; skew 500 ppm
Sep 23 17:25:03 dist-test-slave-ptl9 krb5kdc[17728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184303, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:03.031174 18103 init.cc:377] Logged in from keytab as kudu/127.11.229.3@KRBTEST.COM (short username kudu)
I20260923 17:25:03.032167 18103 webserver.cc:460] Webserver started at http://127.11.229.3:37649/ using document root <none> and password file <none>
I20260923 17:25:03.032768 18103 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:03.032832 18103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:03.033025 18103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:25:03.035178 18103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-2/data/instance:
uuid: "e526effe20244f219db6a9693d5dc88f"
format_stamp: "Formatted at 2026-09-23 17:25:03 on dist-test-slave-ptl9"
server_key: "19cf96573f29fe72c42732e791d49b39"
server_key_iv: "1bd99f40b9b1a259a5f2649ba80edbb9"
server_key_version: "encryptionkey@0"
I20260923 17:25:03.035714 18103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance:
uuid: "e526effe20244f219db6a9693d5dc88f"
format_stamp: "Formatted at 2026-09-23 17:25:03 on dist-test-slave-ptl9"
server_key: "19cf96573f29fe72c42732e791d49b39"
server_key_iv: "1bd99f40b9b1a259a5f2649ba80edbb9"
server_key_version: "encryptionkey@0"
I20260923 17:25:03.039608 18103 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:25:03.042125 18118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:03.043496 18103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:25:03.043599 18103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-2/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-2/wal
uuid: "e526effe20244f219db6a9693d5dc88f"
format_stamp: "Formatted at 2026-09-23 17:25:03 on dist-test-slave-ptl9"
server_key: "19cf96573f29fe72c42732e791d49b39"
server_key_iv: "1bd99f40b9b1a259a5f2649ba80edbb9"
server_key_version: "encryptionkey@0"
I20260923 17:25:03.043687 18103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:03.060138 18103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:03.063156 18103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:03.063313 18103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:03.063850 18103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:25:03.064642 18103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:25:03.064694 18103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:03.064735 18103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:25:03.064756 18103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:03.074267 18103 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.3:39069
I20260923 17:25:03.074290 18231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.3:39069 every 8 connection(s)
I20260923 17:25:03.075210 18103 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
I20260923 17:25:03.083388 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 18103
I20260923 17:25:03.083490 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance
I20260923 17:25:03.083783 12180 external_mini_cluster.cc:1428] Setting key 33e5bc7d1503d458ee0d18cdbbfeb113
Sep 23 17:25:03 dist-test-slave-ptl9 krb5kdc[17728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184303, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:25:03.089583 17812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:03.077141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:50175 (local address 127.11.229.62:42357)
0923 17:25:03.077344 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:03.077349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:03.078225 (+ 876us) server_negotiation.cc:407] Connection header received
0923 17:25:03.079537 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:03.079540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:03.079589 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:03.079667 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:03.080842 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:03.081412 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:03.082503 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:03.082713 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:03.082791 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:03.085804 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:03.085808 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:03.085810 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:03.085836 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:03.087378 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:03.087930 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:03.087934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:03.087936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:03.088019 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:03.088377 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:03.088382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:03.088384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:03.088580 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:03.088686 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:03.089277 (+ 591us) server_negotiation.cc:299] Negotiation successful
0923 17:25:03.089417 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 17:25:03.090468 18234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:03.077585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:42357 (local address 127.11.229.3:50175)
0923 17:25:03.078054 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:03.078098 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:03.079338 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:03.079835 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:03.079842 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:03.080209 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:03.080712 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:03.080725 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:03.081521 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:03.081524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:03.082397 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:03.082407 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:03.082843 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:03.082845 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:03.082985 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:03.083760 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:03.083777 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:03.085628 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:03.087515 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:03.087521 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:03.087539 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:03.087812 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:03.088119 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:03.088122 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:03.088124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:03.088278 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:03.088703 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:03.088711 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:03.088995 (+ 284us) client_negotiation.cc:769] Sending connection context
0923 17:25:03.089218 (+ 223us) client_negotiation.cc:240] Negotiation successful
0923 17:25:03.089609 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":120,"threads_started":1}
I20260923 17:25:03.091610 18232 heartbeater.cc:347] Connected to a master server at 127.11.229.62:42357
I20260923 17:25:03.091878 18232 heartbeater.cc:464] Registering TS with master...
I20260923 17:25:03.092407 18232 heartbeater.cc:511] Master 127.11.229.62:42357 requested a full tablet report, sending...
I20260923 17:25:03.093552 17773 ts_manager.cc:194] Registered new tserver with Master: e526effe20244f219db6a9693d5dc88f (127.11.229.3:39069)
I20260923 17:25:03.094153 17773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.3@KRBTEST.COM'} at 127.11.229.3:50175
I20260923 17:25:03.097812 12180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/krb5kdc/test-admin.keytab.
Sep 23 17:25:03 dist-test-slave-ptl9 krb5kdc[17728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:03.113977 12180 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260923 17:25:03.116945 18239 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:03.115427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43548 (local address 127.11.229.1:44819)
0923 17:25:03.115721 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:03.115737 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:03.115768 (+ 31us) server_negotiation.cc:407] Connection header received
0923 17:25:03.115839 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:03.115846 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:03.116066 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:03.116237 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:03.116746 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43548: BlockingRecv error: recv got EOF from 127.0.0.1:43548 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":66,"threads_started":1}
W20260923 17:25:03.117084 12180 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/krb5kdc/krb5cc' of type FILE
Sep 23 17:25:03 dist-test-slave-ptl9 krb5kdc[17728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:03.118176 12180 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 17:25:03 dist-test-slave-ptl9 krb5kdc[17728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:03.119103 12180 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 17:25:03 dist-test-slave-ptl9 krb5kdc[17728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.1@KRBTEST.COM
I20260923 17:25:03.129522 18239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:03.120033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43560 (local address 127.11.229.1:44819)
0923 17:25:03.120174 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:03.120179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:03.120279 (+ 100us) server_negotiation.cc:407] Connection header received
0923 17:25:03.120447 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:03.120450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:03.120523 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:03.120599 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:03.121141 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:03.122098 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:03.122950 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:03.123361 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:03.123479 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:03.125985 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:03.125995 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:03.126009 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:03.126035 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:03.127653 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:03.128109 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:03.128115 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:03.128116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:03.128206 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:03.128482 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:03.128486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:03.128487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:03.128750 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:03.128967 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:03.129182 (+ 215us) server_negotiation.cc:299] Negotiation successful
0923 17:25:03.129379 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260923 17:25:03.133374 18239 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:03.132463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43574 (local address 127.11.229.1:44819)
0923 17:25:03.132700 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:03.132704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:03.132716 (+ 12us) server_negotiation.cc:407] Connection header received
0923 17:25:03.132748 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:03.132750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:03.132785 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:03.132848 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:03.133260 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43574: BlockingRecv error: recv got EOF from 127.0.0.1:43574 (error 108)
Metrics: {"server-negotiator.queue_time_us":156}
W20260923 17:25:03.133507 12180 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/krb5kdc/krb5cc' of type FILE
Sep 23 17:25:03 dist-test-slave-ptl9 krb5kdc[17728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:03.134536 12180 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 17:25:03 dist-test-slave-ptl9 krb5kdc[17728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:03.135442 12180 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 17:25:03 dist-test-slave-ptl9 krb5kdc[17728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.1@KRBTEST.COM
I20260923 17:25:03.144660 18239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:03.136276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43588 (local address 127.11.229.1:44819)
0923 17:25:03.136405 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:03.136411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:03.136497 (+ 86us) server_negotiation.cc:407] Connection header received
0923 17:25:03.136665 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:03.136668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:03.136715 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:03.136800 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:03.137347 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:03.137870 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:03.138694 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:03.138930 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:03.139043 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:03.141428 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:03.141431 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:03.141433 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:03.141456 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:03.143001 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:03.143425 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:03.143429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:03.143431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:03.143515 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:03.143801 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:03.143806 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:03.143808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:03.143988 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:03.144135 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:03.144373 (+ 238us) server_negotiation.cc:299] Negotiation successful
0923 17:25:03.144489 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260923 17:25:03.147505 18239 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:03.146596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43590 (local address 127.11.229.1:44819)
0923 17:25:03.146735 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:03.146738 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:03.146753 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:25:03.146903 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:03.146905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:03.146941 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:03.147005 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:03.147403 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43590: BlockingRecv error: recv got EOF from 127.0.0.1:43590 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260923 17:25:03.147651 12180 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184271597484-12180-0/krb5kdc/krb5cc' of type FILE
Sep 23 17:25:03 dist-test-slave-ptl9 krb5kdc[17728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:03.148801 12180 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 17:25:03 dist-test-slave-ptl9 krb5kdc[17728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:03.149781 12180 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 17:25:03 dist-test-slave-ptl9 krb5kdc[17728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.1@KRBTEST.COM
I20260923 17:25:03.158368 18239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:03.150560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43594 (local address 127.11.229.1:44819)
0923 17:25:03.150693 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:03.150697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:03.150763 (+ 66us) server_negotiation.cc:407] Connection header received
0923 17:25:03.150905 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:03.150908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:03.150946 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:03.151035 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:03.151534 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:03.152039 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:03.152852 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:03.153089 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:03.153196 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:03.155514 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:03.155517 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:03.155519 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:03.155547 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:03.156991 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:03.157430 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:03.157433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:03.157434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:03.157508 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:03.157722 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:03.157724 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:03.157726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:03.157875 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:03.157963 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:03.158149 (+ 186us) server_negotiation.cc:299] Negotiation successful
0923 17:25:03.158241 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 17:25:03.159457 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 17835
I20260923 17:25:03.166023 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 17969
I20260923 17:25:03.171875 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 18103
I20260923 17:25:03.177300 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 17741
2026-09-23T17:25:03Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2196 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:25:03 dist-test-slave-ptl9 krb5kdc[18252](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:25:03 dist-test-slave-ptl9 krb5kdc[18252](info): set up 2 sockets
Sep 23 17:25:03 dist-test-slave-ptl9 krb5kdc[18252](info): commencing operation
krb5kdc: starting...
W20260923 17:25:04.205817 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:25:04 dist-test-slave-ptl9 krb5kdc[18252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:25:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:25:04Z Disabled control of system clock
WARNING: no policy specified for oryx/127.11.229.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.229.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.229.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.62.keytab.
Entry for principal oryx/127.11.229.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.62.keytab.
I20260923 17:25:04.328322 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:46815
--webserver_interface=127.11.229.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.62.keytab
--principal=oryx/127.11.229.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:44685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:46815
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:25:04.442765 18265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:04.443058 18265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:04.443100 18265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:04.447225 18265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:25:04.447294 18265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:04.447314 18265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:04.447330 18265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:25:04.447347 18265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:25:04.452819 18265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:44685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:46815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.62.keytab
--principal=oryx/127.11.229.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:46815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:25:04.453990 18265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:04.455096 18265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:04.465926 18273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:04.465920 18270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:04.466126 18271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:04.466310 18265 server_base.cc:1034] running on GCE node
I20260923 17:25:04.467293 18265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:04.468914 18265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:04.470132 18265 hybrid_clock.cc:648] HybridClock initialized: now 1790184304470104 us; error 43 us; skew 500 ppm
Sep 23 17:25:04 dist-test-slave-ptl9 krb5kdc[18252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184304, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:04.479839 18265 init.cc:377] Logged in from keytab as oryx/127.11.229.62@KRBTEST.COM (short username oryx)
I20260923 17:25:04.480826 18265 webserver.cc:460] Webserver started at http://127.11.229.62:46401/ using document root <none> and password file <none>
I20260923 17:25:04.481505 18265 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:04.481601 18265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:04.481837 18265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:25:04.483980 18265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "b5536110589147d9b37d53ea18ab3f26"
format_stamp: "Formatted at 2026-09-23 17:25:04 on dist-test-slave-ptl9"
server_key: "e902692594d7c631e81d7dff7861b48b"
server_key_iv: "b3fc2503b93e2a1435538166c5beafb1"
server_key_version: "encryptionkey@0"
I20260923 17:25:04.484593 18265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "b5536110589147d9b37d53ea18ab3f26"
format_stamp: "Formatted at 2026-09-23 17:25:04 on dist-test-slave-ptl9"
server_key: "e902692594d7c631e81d7dff7861b48b"
server_key_iv: "b3fc2503b93e2a1435538166c5beafb1"
server_key_version: "encryptionkey@0"
I20260923 17:25:04.488437 18265 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 17:25:04.490942 18280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:04.492161 18265 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:25:04.492309 18265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "b5536110589147d9b37d53ea18ab3f26"
format_stamp: "Formatted at 2026-09-23 17:25:04 on dist-test-slave-ptl9"
server_key: "e902692594d7c631e81d7dff7861b48b"
server_key_iv: "b3fc2503b93e2a1435538166c5beafb1"
server_key_version: "encryptionkey@0"
I20260923 17:25:04.492424 18265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:04.506641 18265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:04.509723 18265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:04.509936 18265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:04.517280 18265 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:46815
I20260923 17:25:04.517668 18332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:46815 every 8 connection(s)
I20260923 17:25:04.518313 18265 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:25:04.521142 18333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:04.525049 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 18265
I20260923 17:25:04.525189 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:25:04.525525 12180 external_mini_cluster.cc:1428] Setting key c328430fbefdec1bc23757d5524b9ea1
I20260923 17:25:04.527372 18333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26: Bootstrap starting.
I20260923 17:25:04.529713 18333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:04.530471 18333 log.cc:824] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26: Log is configured to *not* fsync() on all Append() calls
I20260923 17:25:04.532598 18333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26: No bootstrap required, opened a new log
Sep 23 17:25:04 dist-test-slave-ptl9 krb5kdc[18252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM
I20260923 17:25:04.537446 18333 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } }
I20260923 17:25:04.537734 18333 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:04.537794 18333 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5536110589147d9b37d53ea18ab3f26, State: Initialized, Role: FOLLOWER
I20260923 17:25:04.538322 18333 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } }
I20260923 17:25:04.538496 18333 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:25:04.538568 18333 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:25:04.538664 18333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:04.539811 18333 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } }
I20260923 17:25:04.540177 18333 leader_election.cc:302] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5536110589147d9b37d53ea18ab3f26; no voters:
I20260923 17:25:04.540506 18333 leader_election.cc:288] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:25:04.540657 18336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:04.526966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44916 (local address 127.11.229.62:46815)
0923 17:25:04.527419 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:04.527435 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:04.527500 (+ 65us) server_negotiation.cc:407] Connection header received
0923 17:25:04.528163 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:04.528180 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:04.528516 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:04.528869 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:04.529740 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:04.530876 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:04.532017 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:04.532482 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:04.532610 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:04.535299 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:04.535308 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:04.535331 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:04.535364 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:04.537389 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:04.537911 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:04.537917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:04.537924 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:04.538062 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:04.538350 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:04.538353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:04.538355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:04.538900 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:04.539101 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:04.539333 (+ 232us) server_negotiation.cc:299] Negotiation successful
0923 17:25:04.539654 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":140,"threads_started":1}
I20260923 17:25:04.540725 18338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:25:04.540952 18338 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [term 1 LEADER]: Becoming Leader. State: Replica: b5536110589147d9b37d53ea18ab3f26, State: Running, Role: LEADER
I20260923 17:25:04.541419 18338 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } }
I20260923 17:25:04.541829 18333 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:25:04.542668 18339 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5536110589147d9b37d53ea18ab3f26. Latest consensus state: current_term: 1 leader_uuid: "b5536110589147d9b37d53ea18ab3f26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } }
I20260923 17:25:04.542785 18339 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [sys.catalog]: This master's current role is: LEADER
I20260923 17:25:04.543432 18340 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5536110589147d9b37d53ea18ab3f26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } }
I20260923 17:25:04.543524 18340 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [sys.catalog]: This master's current role is: LEADER
I20260923 17:25:04.543797 18347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:25:04.547277 18347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:25:04.553354 18347 catalog_manager.cc:1347] Generated new cluster ID: 78a04d62a08a462dab6b3c54b4731f38
I20260923 17:25:04.553427 18347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:25:04.562265 18347 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:25:04.563488 18347 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:25:04.573354 18347 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26: Generated new TSK 0
I20260923 17:25:04.574098 18347 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.11.229.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.229.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.229.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.1.keytab.
Entry for principal oryx/127.11.229.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.1.keytab.
I20260923 17:25:04.609510 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.1:0
--local_ip_for_outbound_sockets=127.11.229.1
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:46815
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.1.keytab
--principal=oryx/127.11.229.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:44685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:25:04.726081 18359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:04.726336 18359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:04.726403 18359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:04.730427 18359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:04.730486 18359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:04.730575 18359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.1
I20260923 17:25:04.735898 18359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:44685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.1.keytab
--principal=oryx/127.11.229.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:46815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.1
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:25:04.736995 18359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:04.738130 18359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:04.748323 18367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:04.748307 18364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:04.748390 18359 server_base.cc:1034] running on GCE node
W20260923 17:25:04.748311 18365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:04.748914 18359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:04.749682 18359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:04.750877 18359 hybrid_clock.cc:648] HybridClock initialized: now 1790184304750860 us; error 37 us; skew 500 ppm
Sep 23 17:25:04 dist-test-slave-ptl9 krb5kdc[18252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184304, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:04.753810 18359 init.cc:377] Logged in from keytab as oryx/127.11.229.1@KRBTEST.COM (short username oryx)
I20260923 17:25:04.754745 18359 webserver.cc:460] Webserver started at http://127.11.229.1:40443/ using document root <none> and password file <none>
I20260923 17:25:04.755349 18359 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:04.755424 18359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:04.755718 18359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:25:04.757979 18359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-0/data/instance:
uuid: "e507acf6b90b45269c0f77c976e485ca"
format_stamp: "Formatted at 2026-09-23 17:25:04 on dist-test-slave-ptl9"
server_key: "74a9f52ebb4b7b1ec000a49416ece6ca"
server_key_iv: "f2d9e102488f71dda0b9e88c3a9c7091"
server_key_version: "encryptionkey@0"
I20260923 17:25:04.758569 18359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance:
uuid: "e507acf6b90b45269c0f77c976e485ca"
format_stamp: "Formatted at 2026-09-23 17:25:04 on dist-test-slave-ptl9"
server_key: "74a9f52ebb4b7b1ec000a49416ece6ca"
server_key_iv: "f2d9e102488f71dda0b9e88c3a9c7091"
server_key_version: "encryptionkey@0"
I20260923 17:25:04.762584 18359 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260923 17:25:04.765038 18374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:04.766443 18359 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 17:25:04.766568 18359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-0/wal
uuid: "e507acf6b90b45269c0f77c976e485ca"
format_stamp: "Formatted at 2026-09-23 17:25:04 on dist-test-slave-ptl9"
server_key: "74a9f52ebb4b7b1ec000a49416ece6ca"
server_key_iv: "f2d9e102488f71dda0b9e88c3a9c7091"
server_key_version: "encryptionkey@0"
I20260923 17:25:04.766690 18359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:04.778352 18359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:04.781531 18359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:04.781738 18359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:04.782359 18359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:25:04.783213 18359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:25:04.783259 18359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:04.783329 18359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:25:04.783360 18359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:04.792348 18359 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.1:36025
I20260923 17:25:04.792418 18487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.1:36025 every 8 connection(s)
I20260923 17:25:04.793416 18359 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
I20260923 17:25:04.796283 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 18359
I20260923 17:25:04.796430 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance
I20260923 17:25:04.796728 12180 external_mini_cluster.cc:1428] Setting key 5e83df0491615134ea2a8ebe3cc6cce0
Sep 23 17:25:04 dist-test-slave-ptl9 krb5kdc[18252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184304, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.11.229.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.229.2@KRBTEST.COM" created.
I20260923 17:25:04.809159 18336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:04.795370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:43817 (local address 127.11.229.62:46815)
0923 17:25:04.795557 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:04.795562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:04.796545 (+ 983us) server_negotiation.cc:407] Connection header received
0923 17:25:04.798391 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:04.798394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:04.798437 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:04.798507 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:04.800233 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:04.801092 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:04.802351 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:04.802594 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:04.802727 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:04.805321 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:04.805329 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:04.805331 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:04.805356 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:04.806875 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:04.807450 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:04.807453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:04.807454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:04.807533 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:04.807897 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:04.807901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:04.807903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:04.808099 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:04.808218 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:04.808797 (+ 579us) server_negotiation.cc:299] Negotiation successful
0923 17:25:04.808974 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 17:25:04.810264 18490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:04.795820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46815 (local address 127.11.229.1:43817)
0923 17:25:04.796390 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:04.796444 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:04.798111 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:04.798712 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:04.798722 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:04.799280 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:04.800008 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:04.800032 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:04.801237 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:04.801239 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:04.802211 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:04.802221 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:04.802776 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:04.802780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:04.802959 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:04.803605 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:04.803623 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:04.805107 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:04.806998 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:04.807004 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:04.807022 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:04.807301 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:04.807638 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:04.807641 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:04.807643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:04.807793 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:04.808294 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:04.808308 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:04.808563 (+ 255us) client_negotiation.cc:769] Sending connection context
0923 17:25:04.808740 (+ 177us) client_negotiation.cc:240] Negotiation successful
0923 17:25:04.809088 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":206,"threads_started":1}
I20260923 17:25:04.811934 18488 heartbeater.cc:347] Connected to a master server at 127.11.229.62:46815
I20260923 17:25:04.812275 18488 heartbeater.cc:464] Registering TS with master...
I20260923 17:25:04.812903 18488 heartbeater.cc:511] Master 127.11.229.62:46815 requested a full tablet report, sending...
I20260923 17:25:04.814808 18297 ts_manager.cc:194] Registered new tserver with Master: e507acf6b90b45269c0f77c976e485ca (127.11.229.1:36025)
I20260923 17:25:04.815986 18297 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.229.1@KRBTEST.COM'} at 127.11.229.1:43817
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.229.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.2.keytab.
Entry for principal oryx/127.11.229.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.2.keytab.
I20260923 17:25:04.827922 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.2:0
--local_ip_for_outbound_sockets=127.11.229.2
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:46815
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.2.keytab
--principal=oryx/127.11.229.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:44685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:25:04.942919 18493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:04.943344 18493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:04.943435 18493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:04.947854 18493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:04.947965 18493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:04.948068 18493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.2
I20260923 17:25:04.953719 18493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:44685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.2.keytab
--principal=oryx/127.11.229.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:46815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.2
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:25:04.954900 18493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:04.956054 18493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:04.966115 18498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:04.966148 18501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:04.966118 18499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:04.966543 18493 server_base.cc:1034] running on GCE node
I20260923 17:25:04.967170 18493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:04.967919 18493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:04.969168 18493 hybrid_clock.cc:648] HybridClock initialized: now 1790184304969148 us; error 32 us; skew 500 ppm
Sep 23 17:25:04 dist-test-slave-ptl9 krb5kdc[18252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184304, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:04.972242 18493 init.cc:377] Logged in from keytab as oryx/127.11.229.2@KRBTEST.COM (short username oryx)
I20260923 17:25:04.973235 18493 webserver.cc:460] Webserver started at http://127.11.229.2:46283/ using document root <none> and password file <none>
I20260923 17:25:04.973883 18493 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:04.973989 18493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:04.974248 18493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:25:04.976353 18493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-1/data/instance:
uuid: "50709fadaa9e44cf830396ddbb1b72a4"
format_stamp: "Formatted at 2026-09-23 17:25:04 on dist-test-slave-ptl9"
server_key: "6fb8dd01dbf121364078ea16feba0206"
server_key_iv: "35405b5b7c85db3d61328b038dcd9f66"
server_key_version: "encryptionkey@0"
I20260923 17:25:04.976944 18493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance:
uuid: "50709fadaa9e44cf830396ddbb1b72a4"
format_stamp: "Formatted at 2026-09-23 17:25:04 on dist-test-slave-ptl9"
server_key: "6fb8dd01dbf121364078ea16feba0206"
server_key_iv: "35405b5b7c85db3d61328b038dcd9f66"
server_key_version: "encryptionkey@0"
I20260923 17:25:04.980840 18493 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260923 17:25:04.983448 18508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:04.984674 18493 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260923 17:25:04.984812 18493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-1/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-1/wal
uuid: "50709fadaa9e44cf830396ddbb1b72a4"
format_stamp: "Formatted at 2026-09-23 17:25:04 on dist-test-slave-ptl9"
server_key: "6fb8dd01dbf121364078ea16feba0206"
server_key_iv: "35405b5b7c85db3d61328b038dcd9f66"
server_key_version: "encryptionkey@0"
I20260923 17:25:04.984931 18493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:05.002388 18493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:05.005589 18493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:05.005790 18493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:05.006398 18493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:25:05.007236 18493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:25:05.007308 18493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:05.007380 18493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:25:05.007434 18493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:05.016525 18493 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.2:36405
I20260923 17:25:05.016553 18621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.2:36405 every 8 connection(s)
I20260923 17:25:05.017468 18493 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
I20260923 17:25:05.025076 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 18493
I20260923 17:25:05.025182 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance
I20260923 17:25:05.025532 12180 external_mini_cluster.cc:1428] Setting key 4592f72bf1db0b1c6a52c03cd490282c
Sep 23 17:25:05 dist-test-slave-ptl9 krb5kdc[18252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184304, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM
I20260923 17:25:05.032572 18336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.019354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:60973 (local address 127.11.229.62:46815)
0923 17:25:05.019519 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:05.019531 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:05.020378 (+ 847us) server_negotiation.cc:407] Connection header received
0923 17:25:05.021896 (+ 1518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:05.021900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:05.021960 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:05.022037 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:05.023227 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.023778 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.024948 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.025233 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.025364 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:05.028223 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:05.028227 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:05.028230 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:05.028261 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:05.030123 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:05.030783 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:05.030785 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:05.030787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:05.030863 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:05.031344 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:05.031347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:05.031349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:05.031593 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:05.031721 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:05.032272 (+ 551us) server_negotiation.cc:299] Negotiation successful
0923 17:25:05.032401 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 17:25:05.033530 18624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.019729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46815 (local address 127.11.229.2:60973)
0923 17:25:05.020221 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:05.020266 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:05.021673 (+ 1407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:05.022219 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:05.022226 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:05.022584 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:05.023068 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:05.023080 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.023915 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.023919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:05.024813 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:05.024822 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.025405 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.025410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:05.025596 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:05.026313 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:05.026338 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:05.028033 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:05.030262 (+ 2229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:05.030271 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:05.030291 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:05.030682 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:05.030958 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:05.030998 (+ 40us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:05.031003 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:05.031203 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:05.031720 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:05.031728 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:05.032008 (+ 280us) client_negotiation.cc:769] Sending connection context
0923 17:25:05.032227 (+ 219us) client_negotiation.cc:240] Negotiation successful
0923 17:25:05.032588 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":120,"threads_started":1}
I20260923 17:25:05.034942 18622 heartbeater.cc:347] Connected to a master server at 127.11.229.62:46815
I20260923 17:25:05.035218 18622 heartbeater.cc:464] Registering TS with master...
I20260923 17:25:05.035748 18622 heartbeater.cc:511] Master 127.11.229.62:46815 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.11.229.3@KRBTEST.COM; defaulting to no policy
I20260923 17:25:05.036944 18297 ts_manager.cc:194] Registered new tserver with Master: 50709fadaa9e44cf830396ddbb1b72a4 (127.11.229.2:36405)
I20260923 17:25:05.037601 18297 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.229.2@KRBTEST.COM'} at 127.11.229.2:60973
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.229.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.229.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.3.keytab.
Entry for principal oryx/127.11.229.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.3.keytab.
I20260923 17:25:05.056303 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.3:0
--local_ip_for_outbound_sockets=127.11.229.3
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:46815
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.3.keytab
--principal=oryx/127.11.229.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:44685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:25:05.170217 18627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:05.170490 18627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:05.170536 18627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:05.174779 18627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:05.174850 18627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:05.174913 18627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.3
I20260923 17:25:05.180286 18627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:44685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.3.keytab
--principal=oryx/127.11.229.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:46815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.3
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:25:05.181453 18627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:05.182626 18627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 17:25:05.192801 18627 server_base.cc:1034] running on GCE node
W20260923 17:25:05.192734 18633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:05.192723 18635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:05.192719 18632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:05.193490 18627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:05.194129 18627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:05.195307 18627 hybrid_clock.cc:648] HybridClock initialized: now 1790184305195288 us; error 30 us; skew 500 ppm
Sep 23 17:25:05 dist-test-slave-ptl9 krb5kdc[18252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184305, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:05.198347 18627 init.cc:377] Logged in from keytab as oryx/127.11.229.3@KRBTEST.COM (short username oryx)
I20260923 17:25:05.199260 18627 webserver.cc:460] Webserver started at http://127.11.229.3:33833/ using document root <none> and password file <none>
I20260923 17:25:05.199859 18627 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:05.199921 18627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:05.200176 18627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:25:05.202420 18627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-2/data/instance:
uuid: "59e579024b004c1ebea39d11c0ce6461"
format_stamp: "Formatted at 2026-09-23 17:25:05 on dist-test-slave-ptl9"
server_key: "5ec131cf45ea2f9fd6129d62700997ff"
server_key_iv: "f5db46fdb2bacbfca77a5401d0ac3452"
server_key_version: "encryptionkey@0"
I20260923 17:25:05.203011 18627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance:
uuid: "59e579024b004c1ebea39d11c0ce6461"
format_stamp: "Formatted at 2026-09-23 17:25:05 on dist-test-slave-ptl9"
server_key: "5ec131cf45ea2f9fd6129d62700997ff"
server_key_iv: "f5db46fdb2bacbfca77a5401d0ac3452"
server_key_version: "encryptionkey@0"
I20260923 17:25:05.206929 18627 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:25:05.209641 18642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:05.210896 18627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 17:25:05.211045 18627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-2/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-2/wal
uuid: "59e579024b004c1ebea39d11c0ce6461"
format_stamp: "Formatted at 2026-09-23 17:25:05 on dist-test-slave-ptl9"
server_key: "5ec131cf45ea2f9fd6129d62700997ff"
server_key_iv: "f5db46fdb2bacbfca77a5401d0ac3452"
server_key_version: "encryptionkey@0"
I20260923 17:25:05.211179 18627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:05.239470 18627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:05.243249 18627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:05.243480 18627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:05.244113 18627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:25:05.244964 18627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:25:05.245036 18627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:05.245114 18627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:25:05.245170 18627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:05.254145 18627 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.3:35491
I20260923 17:25:05.254223 18755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.3:35491 every 8 connection(s)
I20260923 17:25:05.255090 18627 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
I20260923 17:25:05.263044 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 18627
I20260923 17:25:05.263291 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance
I20260923 17:25:05.263576 12180 external_mini_cluster.cc:1428] Setting key 74eb1be56fc005b5fc38b7485a23bdd5
Sep 23 17:25:05 dist-test-slave-ptl9 krb5kdc[18252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184305, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.3@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM
I20260923 17:25:05.268648 18336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.256986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:36367 (local address 127.11.229.62:46815)
0923 17:25:05.257142 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:05.257146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:05.258208 (+ 1062us) server_negotiation.cc:407] Connection header received
0923 17:25:05.259038 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:05.259040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:05.259087 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:05.259173 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:05.260355 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.260883 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.262115 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.262345 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.262432 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:05.265306 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:05.265310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:05.265315 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:05.265344 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:05.266796 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:05.267270 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:05.267272 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:05.267273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:05.267349 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:05.267662 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:05.267665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:05.267666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:05.267835 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:05.267912 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:05.268422 (+ 510us) server_negotiation.cc:299] Negotiation successful
0923 17:25:05.268527 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:25:05.269533 18758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.257399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46815 (local address 127.11.229.3:36367)
0923 17:25:05.258047 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:05.258087 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:05.258855 (+ 768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:05.259335 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:05.259342 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:05.259714 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:05.260189 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:05.260202 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.261030 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.261032 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:05.262000 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:05.262010 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.262482 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.262484 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:05.262634 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:05.263291 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:05.263316 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:05.265099 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:05.266906 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:05.266911 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:05.266924 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:05.267181 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:05.267432 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:05.267434 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:05.267435 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:05.267568 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:05.267925 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:05.267930 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:05.268171 (+ 241us) client_negotiation.cc:769] Sending connection context
0923 17:25:05.268352 (+ 181us) client_negotiation.cc:240] Negotiation successful
0923 17:25:05.268617 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":115,"threads_started":1}
I20260923 17:25:05.270752 18756 heartbeater.cc:347] Connected to a master server at 127.11.229.62:46815
I20260923 17:25:05.271020 18756 heartbeater.cc:464] Registering TS with master...
I20260923 17:25:05.271624 18756 heartbeater.cc:511] Master 127.11.229.62:46815 requested a full tablet report, sending...
I20260923 17:25:05.272708 18297 ts_manager.cc:194] Registered new tserver with Master: 59e579024b004c1ebea39d11c0ce6461 (127.11.229.3:35491)
I20260923 17:25:05.273370 18297 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.229.3@KRBTEST.COM'} at 127.11.229.3:36367
I20260923 17:25:05.277637 12180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.11.229.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.229.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.229.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.61.keytab.
Entry for principal oryx/127.11.229.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.61.keytab.
I20260923 17:25:05.304251 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.61:41129
--webserver_interface=127.11.229.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.61.keytab
--principal=oryx/127.11.229.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:44685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:46815,127.11.229.61:41129
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:25:05.420735 18761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:05.421000 18761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:05.421039 18761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:05.425079 18761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:25:05.425154 18761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:05.425174 18761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:05.425191 18761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:25:05.425235 18761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:25:05.430722 18761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:44685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:46815,127.11.229.61:41129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.61.keytab
--principal=oryx/127.11.229.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.61:41129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.229.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
W20260923 17:25:05.430953 18761 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.11.229.62:46815,127.11.229.61:41129'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260923 17:25:05.431869 18761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:05.432951 18761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:05.441639 18769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:05.441664 18767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:05.441660 18766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:05.441896 18761 server_base.cc:1034] running on GCE node
I20260923 17:25:05.442658 18761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:05.443914 18761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:05.445137 18761 hybrid_clock.cc:648] HybridClock initialized: now 1790184305445120 us; error 31 us; skew 500 ppm
Sep 23 17:25:05 dist-test-slave-ptl9 krb5kdc[18252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184305, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:05.448115 18761 init.cc:377] Logged in from keytab as oryx/127.11.229.61@KRBTEST.COM (short username oryx)
I20260923 17:25:05.449064 18761 webserver.cc:460] Webserver started at http://127.11.229.61:35749/ using document root <none> and password file <none>
I20260923 17:25:05.449713 18761 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:05.449805 18761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:05.450039 18761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:25:05.452109 18761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-1/data/instance:
uuid: "23b480ef1799433abb57f326cf3078be"
format_stamp: "Formatted at 2026-09-23 17:25:05 on dist-test-slave-ptl9"
server_key: "545ba34b6b7eb7750c53e640dad90816"
server_key_iv: "74ebdafafcc8425b33650cd697e83a56"
server_key_version: "encryptionkey@0"
I20260923 17:25:05.452684 18761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-1/wal/instance:
uuid: "23b480ef1799433abb57f326cf3078be"
format_stamp: "Formatted at 2026-09-23 17:25:05 on dist-test-slave-ptl9"
server_key: "545ba34b6b7eb7750c53e640dad90816"
server_key_iv: "74ebdafafcc8425b33650cd697e83a56"
server_key_version: "encryptionkey@0"
I20260923 17:25:05.456461 18761 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260923 17:25:05.459064 18776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:05.460448 18761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:25:05.460642 18761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-1/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-1/wal
uuid: "23b480ef1799433abb57f326cf3078be"
format_stamp: "Formatted at 2026-09-23 17:25:05 on dist-test-slave-ptl9"
server_key: "545ba34b6b7eb7750c53e640dad90816"
server_key_iv: "74ebdafafcc8425b33650cd697e83a56"
server_key_version: "encryptionkey@0"
I20260923 17:25:05.460776 18761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:05.477867 18761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:05.481709 18761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:05.481885 18761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:05.489558 18761 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.61:41129
I20260923 17:25:05.489846 18828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.61:41129 every 8 connection(s)
I20260923 17:25:05.490686 18761 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-1/data/info.pb
I20260923 17:25:05.493666 18829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:05.497856 18829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:25:05.501003 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 18761
Sep 23 17:25:05 dist-test-slave-ptl9 krb5kdc[18252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184305, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.61@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM
Sep 23 17:25:05 dist-test-slave-ptl9 krb5kdc[18252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.229.61@KRBTEST.COM
I20260923 17:25:05.512956 18336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.499818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44924 (local address 127.11.229.62:46815)
0923 17:25:05.500013 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:05.500017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:05.500958 (+ 941us) server_negotiation.cc:407] Connection header received
0923 17:25:05.502293 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:05.502299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:05.502368 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:05.502419 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:05.503702 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.504287 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.505482 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.505752 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.505874 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:05.508540 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:05.508544 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:05.508546 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:05.508574 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:05.510465 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:05.511097 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:05.511102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:05.511104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:05.511207 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:05.511619 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:05.511622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:05.511624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:05.511961 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:05.512052 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:05.512575 (+ 523us) server_negotiation.cc:299] Negotiation successful
0923 17:25:05.512725 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260923 17:25:05.514235 18831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.500196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46815 (local address 127.0.0.1:44924)
0923 17:25:05.500736 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:05.500812 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:05.501998 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:05.502773 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:05.502792 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:05.503157 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:05.503542 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:05.503556 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.504437 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.504441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:05.505347 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:05.505360 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.505914 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.505919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:05.506100 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:05.506720 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:05.506745 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:05.508335 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:05.510596 (+ 2261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:05.510605 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:05.510618 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:05.510980 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:05.511333 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:05.511337 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:05.511339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:05.511505 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:05.512074 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:05.512080 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:05.512316 (+ 236us) client_negotiation.cc:769] Sending connection context
0923 17:25:05.512505 (+ 189us) client_negotiation.cc:240] Negotiation successful
0923 17:25:05.512872 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":4,"spinlock_wait_cycles":2816,"thread_start_us":119,"threads_started":1}
I20260923 17:25:05.514235 18833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.501985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52136 (local address 127.11.229.61:41129)
0923 17:25:05.502347 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:05.502360 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:05.502415 (+ 55us) server_negotiation.cc:407] Connection header received
0923 17:25:05.502573 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:05.502587 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:05.502765 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:05.502948 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:05.503799 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.504665 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.505565 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.506092 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.506215 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:05.508947 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:05.508953 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:05.508965 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:05.508986 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:05.510810 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:05.511342 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:05.511350 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:05.511352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:05.511505 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:05.511835 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:05.511837 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:05.511838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:05.512316 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:05.512509 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:05.512785 (+ 276us) server_negotiation.cc:299] Negotiation successful
0923 17:25:05.513234 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":260,"spinlock_wait_cycles":8704,"thread_start_us":154,"threads_started":1}
I20260923 17:25:05.515645 18829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41129 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:25:05 dist-test-slave-ptl9 krb5kdc[18252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184305, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.61@KRBTEST.COM for oryx/127.11.229.61@KRBTEST.COM
I20260923 17:25:05.525224 18831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.516213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:41129 (local address 127.0.0.1:52140)
0923 17:25:05.516332 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:05.516344 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:05.516433 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:05.516795 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:05.516797 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:05.516926 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:05.517010 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:05.517056 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.518125 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.518128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:05.518664 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:05.518671 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.519885 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.519888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:05.519964 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:05.520471 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:05.520482 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:05.521975 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:05.523833 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:05.523836 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:05.523837 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:05.524060 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:05.524356 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:05.524359 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:05.524360 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:05.524440 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:05.524840 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:05.524842 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:05.524915 (+ 73us) client_negotiation.cc:769] Sending connection context
0923 17:25:05.524993 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 17:25:05.525092 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260923 17:25:05.525404 18833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.516342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52140 (local address 127.11.229.61:41129)
0923 17:25:05.516439 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:05.516443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:05.516457 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:25:05.516514 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:05.516517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:05.516556 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:05.516642 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:05.517181 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.518010 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.518771 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.519760 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.519878 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:05.522110 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:05.522115 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:05.522118 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:05.522144 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:05.523709 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:05.524156 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:05.524160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:05.524162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:05.524248 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:05.524531 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:05.524534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:05.524536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:05.524747 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:05.524847 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:05.525074 (+ 227us) server_negotiation.cc:299] Negotiation successful
0923 17:25:05.525191 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 17:25:05.529210 18829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be: Bootstrap starting.
I20260923 17:25:05.532239 18829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:05.533190 18829 log.cc:824] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be: Log is configured to *not* fsync() on all Append() calls
I20260923 17:25:05.535365 18829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be: No bootstrap required, opened a new log
I20260923 17:25:05.538668 18829 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41129 } }
I20260923 17:25:05.538880 18829 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:05.538916 18829 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b480ef1799433abb57f326cf3078be, State: Initialized, Role: FOLLOWER
I20260923 17:25:05.539330 18829 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41129 } }
I20260923 17:25:05.539742 18835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41129 } } }
I20260923 17:25:05.539883 18835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:25:05.540561 18829 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [sys.catalog]: configured and running, proceeding with master startup.
W20260923 17:25:05.545797 18846 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:25:05.545892 18846 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:25:05.548079 18761 master_runner.cc:428] Detected that this master 23b480ef1799433abb57f326cf3078be is joining an existing cluster
I20260923 17:25:05.548147 18761 master_runner.cc:432] Initiating AddMaster RPC to add 127.11.229.61:41129
I20260923 17:25:05.559580 18833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.550721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52150 (local address 127.11.229.61:41129)
0923 17:25:05.550838 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:05.550843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:05.551188 (+ 345us) server_negotiation.cc:407] Connection header received
0923 17:25:05.551336 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:05.551341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:05.551382 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:05.551463 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:05.551998 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.552806 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.554258 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.554638 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.554717 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:05.556041 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:05.556048 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:05.556053 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:05.556084 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:05.557987 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:05.558399 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:05.558404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:05.558406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:05.558489 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:05.558782 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:05.558785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:05.558787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:05.558949 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:05.559048 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:05.559298 (+ 250us) server_negotiation.cc:299] Negotiation successful
0923 17:25:05.559443 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":31}
I20260923 17:25:05.559787 18336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.550730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44930 (local address 127.11.229.62:46815)
0923 17:25:05.550850 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:05.550857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:05.551098 (+ 241us) server_negotiation.cc:407] Connection header received
0923 17:25:05.551300 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:05.551306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:05.551358 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:05.551414 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:05.552168 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.552937 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.553884 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.554124 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.554245 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:05.555472 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:05.555476 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:05.555479 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:05.555504 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:05.557534 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:05.558197 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:05.558203 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:05.558206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:05.558308 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:05.558687 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:05.558691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:05.558693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:05.558934 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:05.559019 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:05.559442 (+ 423us) server_negotiation.cc:299] Negotiation successful
0923 17:25:05.559613 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 17:25:05.559593 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.550664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:41129 (local address 127.0.0.1:52150)
0923 17:25:05.551095 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:05.551107 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:05.551235 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:05.551487 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:05.551490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:05.551769 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:05.551887 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:05.551893 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.552950 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.552954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:05.554117 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:05.554129 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.554861 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.554864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:05.554973 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:05.555448 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:05.555465 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:05.555907 (+ 442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:05.558092 (+ 2185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:05.558095 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:05.558099 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:05.558301 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:05.558595 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:05.558598 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:05.558600 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:05.558685 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:05.559038 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:05.559041 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:05.559120 (+ 79us) client_negotiation.cc:769] Sending connection context
0923 17:25:05.559244 (+ 124us) client_negotiation.cc:240] Negotiation successful
0923 17:25:05.559391 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":144,"thread_start_us":72,"threads_started":1}
I20260923 17:25:05.560348 18852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.550665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46815 (local address 127.0.0.1:44930)
0923 17:25:05.551007 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:05.551022 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:05.551175 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:05.551593 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:05.551597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:05.551886 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:05.552024 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:05.552033 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.553049 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.553054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:05.553760 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:05.553770 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.554247 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.554250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:05.554341 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:05.554888 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:05.554900 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:05.555304 (+ 404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:05.557658 (+ 2354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:05.557662 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:05.557665 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:05.558059 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:05.558436 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:05.558439 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:05.558441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:05.558553 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:05.559102 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:05.559105 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:05.559218 (+ 113us) client_negotiation.cc:769] Sending connection context
0923 17:25:05.560014 (+ 796us) client_negotiation.cc:240] Negotiation successful
0923 17:25:05.560161 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":111,"threads_started":1}
Sep 23 17:25:05 dist-test-slave-ptl9 krb5kdc[18252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184304, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.62@KRBTEST.COM for oryx/127.11.229.61@KRBTEST.COM
WARNING: no policy specified for oryx/127.11.229.60@KRBTEST.COM; defaulting to no policy
I20260923 17:25:05.577427 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.565534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:41129 (local address 127.0.0.1:52166)
0923 17:25:05.565938 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:05.566000 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:05.566146 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:05.566502 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:05.566509 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:05.567039 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:05.567189 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:05.567198 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.568346 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.568350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:05.569499 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:05.569511 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.570218 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.570223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:05.570360 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:05.570963 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:05.570980 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:05.573070 (+ 2090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:05.575305 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:05.575318 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:05.575329 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:05.575659 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:05.575996 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:05.576000 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:05.576004 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:05.576135 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:05.576645 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:05.576652 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:05.576820 (+ 168us) client_negotiation.cc:769] Sending connection context
0923 17:25:05.577002 (+ 182us) client_negotiation.cc:240] Negotiation successful
0923 17:25:05.577208 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":105,"threads_started":1}
I20260923 17:25:05.577426 18833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.565675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52166 (local address 127.11.229.61:41129)
0923 17:25:05.565784 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:05.565788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:05.566091 (+ 303us) server_negotiation.cc:407] Connection header received
0923 17:25:05.566300 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:05.566302 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:05.566361 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:05.566435 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:05.567337 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.568192 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.569647 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.570092 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.570217 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:05.573241 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:05.573246 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:05.573249 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:05.573303 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:05.575170 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:05.575780 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:05.575785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:05.575787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:05.575881 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:05.576259 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:05.576262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:05.576264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:05.576509 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:05.576614 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:05.577074 (+ 460us) server_negotiation.cc:299] Negotiation successful
0923 17:25:05.577205 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 17:25:05.579166 18297 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.11.229.61:41129: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "23b480ef1799433abb57f326cf3078be"
member_type: NON_VOTER
last_known_addr {
host: "127.11.229.61"
port: 41129
}
attrs {
promote: true
}
}
dest_uuid: "b5536110589147d9b37d53ea18ab3f26"
cas_config_opid_index: -1
I20260923 17:25:05.579890 18297 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: NON_VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.229.60@KRBTEST.COM" created.
I20260923 17:25:05.582010 18857 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 23b480ef1799433abb57f326cf3078be (127.11.229.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: NON_VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: true } } }
I20260923 17:25:05.582710 18803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:05.583890 18857 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.11.229.61:41129
I20260923 17:25:05.584167 18803 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [term 1 FOLLOWER]: Refusing update from remote peer b5536110589147d9b37d53ea18ab3f26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260923 17:25:05.584208 18857 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b5536110589147d9b37d53ea18ab3f26" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: NON_VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: true } } }
I20260923 17:25:05.584290 18857 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [sys.catalog]: This master's current role is: LEADER
I20260923 17:25:05.584445 18761 master.cc:537] Master@127.11.229.61:41129 shutting down...
I20260923 17:25:05.584878 18858 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: NON_VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 17:25:05.586930 18281 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260923 17:25:05.590119 18761 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:25:05.590440 18761 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:25:05.590569 18761 tablet_replica.cc:330] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be: stopping tablet replica
I20260923 17:25:05.595871 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.595499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:41129 (local address 127.0.0.1:52168)
0923 17:25:05.595685 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:05.595758 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.61:41129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260923 17:25:05.596060 18281 consensus_peers.cc:487] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 -> Peer 23b480ef1799433abb57f326cf3078be (127.11.229.61:41129): Couldn't send request to peer 23b480ef1799433abb57f326cf3078be. Status: Network error: Client connection negotiation failed: client connection to 127.11.229.61:41129: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260923 17:25:05.596184 18761 master.cc:552] Master@127.11.229.61:41129 shutdown complete.
I20260923 17:25:05.596230 18761 master_runner.cc:305] Clearing existing system tablet
I20260923 17:25:05.596510 18761 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:05.596561 18761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:05.598385 18761 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 17:25:05.599319 18863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:05.599550 18761 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:05.599706 18761 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 17:25:05.599752 18761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-1/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-1/wal
uuid: "23b480ef1799433abb57f326cf3078be"
format_stamp: "Formatted at 2026-09-23 17:25:05 on dist-test-slave-ptl9"
server_key: "545ba34b6b7eb7750c53e640dad90816"
server_key_iv: "74ebdafafcc8425b33650cd697e83a56"
server_key_version: "encryptionkey@0"
I20260923 17:25:05.601958 18761 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be: Deleting tablet data with delete state TABLET_DATA_DELETED
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.229.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.60.keytab.
Entry for principal oryx/127.11.229.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.60.keytab.
I20260923 17:25:05.604249 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.60:45961
--webserver_interface=127.11.229.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.60.keytab
--principal=oryx/127.11.229.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:44685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:46815,127.11.229.61:41129,127.11.229.60:45961
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:25:05.604347 18761 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 17:25:05.604410 18761 log.cc:1196] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260923 17:25:05.604583 18761 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be: Deleting consensus metadata
I20260923 17:25:05.604766 18761 master_runner.cc:315] Copying system tablet from 127.11.229.62:46815
I20260923 17:25:05.606025 18761 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be: tablet copy: Beginning tablet copy session from remote peer at address 127.11.229.62:46815
I20260923 17:25:05.614765 18870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.606713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46815 (local address 127.0.0.1:44936)
0923 17:25:05.607212 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:05.607226 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:05.607376 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:05.607668 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:05.607671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:05.607880 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:05.607986 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:05.607992 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.608775 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.608780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:05.609518 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:05.609528 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.610061 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.610065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:05.610171 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:05.610736 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:05.610750 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:05.611078 (+ 328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:05.612898 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:05.612901 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:05.612903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:05.613128 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:05.613466 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:05.613469 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:05.613471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:05.613564 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:05.613965 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:05.613968 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:05.614101 (+ 133us) client_negotiation.cc:769] Sending connection context
0923 17:25:05.614358 (+ 257us) client_negotiation.cc:240] Negotiation successful
0923 17:25:05.614558 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":144,"threads_started":1}
I20260923 17:25:05.614756 18336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.606810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44936 (local address 127.11.229.62:46815)
0923 17:25:05.606992 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:05.607000 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:05.607322 (+ 322us) server_negotiation.cc:407] Connection header received
0923 17:25:05.607503 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:05.607506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:05.607562 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:05.607659 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:05.608110 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.608649 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.609641 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.609936 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.610054 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:05.611235 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:05.611238 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:05.611241 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:05.611269 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:05.612770 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:05.613236 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:05.613239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:05.613241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:05.613363 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:05.613662 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:05.613665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:05.613667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:05.613865 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:05.613976 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:05.614433 (+ 457us) server_negotiation.cc:299] Negotiation successful
0923 17:25:05.614559 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 17:25:05.615844 18317 tablet_copy_service.cc:140] P b5536110589147d9b37d53ea18ab3f26: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 23b480ef1799433abb57f326cf3078be ({username='oryx', principal='oryx/127.11.229.61@KRBTEST.COM'} at 127.0.0.1:44936)
I20260923 17:25:05.616106 18317 tablet_copy_service.cc:161] P b5536110589147d9b37d53ea18ab3f26: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 23b480ef1799433abb57f326cf3078be at {username='oryx', principal='oryx/127.11.229.61@KRBTEST.COM'} at 127.0.0.1:44936: session id = 23b480ef1799433abb57f326cf3078be-00000000000000000000000000000000
I20260923 17:25:05.617856 18317 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 17:25:05.619518 18761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 17:25:05.621803 18761 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be: tablet copy: Starting download of 0 data blocks...
I20260923 17:25:05.622038 18761 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be: tablet copy: Starting download of 1 WAL segments...
I20260923 17:25:05.623893 18761 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 17:25:05.625644 18317 tablet_copy_service.cc:342] P b5536110589147d9b37d53ea18ab3f26: Request end of tablet copy session 23b480ef1799433abb57f326cf3078be-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.11.229.61@KRBTEST.COM'} at 127.0.0.1:44936
I20260923 17:25:05.625778 18317 tablet_copy_service.cc:433] P b5536110589147d9b37d53ea18ab3f26: ending tablet copy session 23b480ef1799433abb57f326cf3078be-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 23b480ef1799433abb57f326cf3078be
I20260923 17:25:05.633477 18761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:05.635238 18871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:05.635300 18874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:05.635808 18761 server_base.cc:1034] running on GCE node
W20260923 17:25:05.637409 18872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:05.637977 18761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:05.638324 18761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:05.639464 18761 hybrid_clock.cc:648] HybridClock initialized: now 1790184305639455 us; error 34 us; skew 500 ppm
Sep 23 17:25:05 dist-test-slave-ptl9 krb5kdc[18252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184305, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:05.640978 18761 init.cc:377] Logged in from keytab as oryx/127.11.229.61@KRBTEST.COM (short username oryx)
I20260923 17:25:05.641523 18761 webserver.cc:460] Webserver started at http://127.11.229.61:38141/ using document root <none> and password file <none>
I20260923 17:25:05.641724 18761 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:05.641794 18761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:05.643158 18761 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 17:25:05.644189 18881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:05.644436 18761 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 17:25:05.644533 18761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-1/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-1/wal
uuid: "23b480ef1799433abb57f326cf3078be"
format_stamp: "Formatted at 2026-09-23 17:25:05 on dist-test-slave-ptl9"
server_key: "545ba34b6b7eb7750c53e640dad90816"
server_key_iv: "74ebdafafcc8425b33650cd697e83a56"
server_key_version: "encryptionkey@0"
I20260923 17:25:05.644640 18761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:05.649842 18761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:05.651866 18761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:05.656833 18761 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.61:41129
I20260923 17:25:05.657413 18933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.61:41129 every 8 connection(s)
I20260923 17:25:05.657450 18761 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-1/data/info.pb
I20260923 17:25:05.659103 18934 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:25:05.660058 18934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be: Bootstrap starting.
I20260923 17:25:05.668692 18934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:25:05.669003 18934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be: Bootstrap complete.
I20260923 17:25:05.669570 18934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: NON_VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: true } }
I20260923 17:25:05.669750 18934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 23b480ef1799433abb57f326cf3078be, State: Initialized, Role: LEARNER
I20260923 17:25:05.669885 18934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: NON_VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: true } }
I20260923 17:25:05.670243 18937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: NON_VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: true } } }
I20260923 17:25:05.670425 18937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [sys.catalog]: This master's current role is: LEARNER
I20260923 17:25:05.670521 18934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:25:05.673740 18948 catalog_manager.cc:1259] Loaded cluster ID: 78a04d62a08a462dab6b3c54b4731f38
I20260923 17:25:05.673820 18948 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be: loading cluster ID for follower catalog manager: success
I20260923 17:25:05.675362 18948 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be: acquiring CA information for follower catalog manager: success
I20260923 17:25:05.676493 18948 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260923 17:25:05.759541 18868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:05.759850 18868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:05.759949 18868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:05.764112 18868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:25:05.764184 18868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:05.764220 18868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:05.764277 18868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:25:05.764312 18868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:25:05.769644 18868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:44685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:46815,127.11.229.61:41129,127.11.229.60:45961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.60.keytab
--principal=oryx/127.11.229.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.60:45961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.229.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:25:05.770891 18868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:05.772118 18868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:05.780923 18954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:05.781183 18956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:05.781195 18953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:05.781354 18868 server_base.cc:1034] running on GCE node
I20260923 17:25:05.782033 18868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:05.783205 18868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:05.784399 18868 hybrid_clock.cc:648] HybridClock initialized: now 1790184305784382 us; error 29 us; skew 500 ppm
Sep 23 17:25:05 dist-test-slave-ptl9 krb5kdc[18252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184305, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:05.787209 18868 init.cc:377] Logged in from keytab as oryx/127.11.229.60@KRBTEST.COM (short username oryx)
I20260923 17:25:05.788134 18868 webserver.cc:460] Webserver started at http://127.11.229.60:40167/ using document root <none> and password file <none>
I20260923 17:25:05.788803 18868 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:05.788895 18868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:05.789135 18868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:25:05.791365 18868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-2/data/instance:
uuid: "b163f7eb24d54419aa3e1337a7c013ae"
format_stamp: "Formatted at 2026-09-23 17:25:05 on dist-test-slave-ptl9"
server_key: "d8df0e242c0583185e6a84b44d31d071"
server_key_iv: "a74e29289ee9fa4295ea50172a653b60"
server_key_version: "encryptionkey@0"
I20260923 17:25:05.792032 18868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-2/wal/instance:
uuid: "b163f7eb24d54419aa3e1337a7c013ae"
format_stamp: "Formatted at 2026-09-23 17:25:05 on dist-test-slave-ptl9"
server_key: "d8df0e242c0583185e6a84b44d31d071"
server_key_iv: "a74e29289ee9fa4295ea50172a653b60"
server_key_version: "encryptionkey@0"
I20260923 17:25:05.796139 18868 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:25:05.798925 18963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:05.800143 18868 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:25:05.800298 18868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-2/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-2/wal
uuid: "b163f7eb24d54419aa3e1337a7c013ae"
format_stamp: "Formatted at 2026-09-23 17:25:05 on dist-test-slave-ptl9"
server_key: "d8df0e242c0583185e6a84b44d31d071"
server_key_iv: "a74e29289ee9fa4295ea50172a653b60"
server_key_version: "encryptionkey@0"
I20260923 17:25:05.800408 18868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:05.818873 18488 heartbeater.cc:503] Master 127.11.229.62:46815 was elected leader, sending a full tablet report...
I20260923 17:25:05.826881 18868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:05.830355 18868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:05.830583 18868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:05.837863 18868 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.60:45961
I20260923 17:25:05.838165 19015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.60:45961 every 8 connection(s)
I20260923 17:25:05.838936 18868 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-2/data/info.pb
I20260923 17:25:05.841907 19016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:05.842617 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 18868
I20260923 17:25:05.847040 19016 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:25:05 dist-test-slave-ptl9 krb5kdc[18252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.229.60@KRBTEST.COM
Sep 23 17:25:05 dist-test-slave-ptl9 krb5kdc[18252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184305, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.60@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM
I20260923 17:25:05.858026 19018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.844153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42774 (local address 127.11.229.60:45961)
0923 17:25:05.844605 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:05.844623 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:05.844660 (+ 37us) server_negotiation.cc:407] Connection header received
0923 17:25:05.845588 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:05.845607 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:05.845975 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:05.846380 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:05.847221 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.848274 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.849327 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.849804 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.849981 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:05.852697 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:05.852703 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:05.852712 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:05.852738 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:05.854738 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:05.855239 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:05.855244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:05.855249 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:05.855352 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:05.855664 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:05.855668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:05.855670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:05.856058 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:05.856253 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:05.856581 (+ 328us) server_negotiation.cc:299] Negotiation successful
0923 17:25:05.856924 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":134,"threads_started":1}
I20260923 17:25:05.858418 18336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.848840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44950 (local address 127.11.229.62:46815)
0923 17:25:05.849420 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:05.849423 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:05.849436 (+ 13us) server_negotiation.cc:407] Connection header received
0923 17:25:05.849475 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:05.849477 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:05.849515 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:05.849610 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:05.850292 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.850802 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.851850 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.852071 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.852199 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:05.854984 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:05.854988 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:05.854991 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:05.855023 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:05.856485 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:05.857024 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:05.857028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:05.857030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:05.857111 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:05.857494 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:05.857498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:05.857500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:05.857682 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:05.857790 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:05.858110 (+ 320us) server_negotiation.cc:299] Negotiation successful
0923 17:25:05.858257 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484}
I20260923 17:25:05.858574 19019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.848664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46815 (local address 127.0.0.1:44950)
0923 17:25:05.848986 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:05.849033 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:05.849148 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:05.849682 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:05.849690 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:05.850033 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:05.850162 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:05.850168 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.850942 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.850945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:05.851707 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:05.851718 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.852252 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.852256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:05.852381 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:05.852946 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:05.852984 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:05.854844 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:05.856637 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:05.856648 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:05.856662 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:05.856915 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:05.857213 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:05.857218 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:05.857220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:05.857396 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:05.857790 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:05.857798 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:05.857913 (+ 115us) client_negotiation.cc:769] Sending connection context
0923 17:25:05.858053 (+ 140us) client_negotiation.cc:240] Negotiation successful
0923 17:25:05.858347 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":98,"threads_started":1}
I20260923 17:25:05.860023 19016 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41129 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:25:05 dist-test-slave-ptl9 krb5kdc[18252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184305, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.60@KRBTEST.COM for oryx/127.11.229.61@KRBTEST.COM
I20260923 17:25:05.868162 19019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.860539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:41129 (local address 127.0.0.1:52170)
0923 17:25:05.860688 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:05.860699 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:05.860762 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:05.861139 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:05.861141 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:05.861338 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:05.861445 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:05.861451 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.862383 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.862386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:05.862932 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:05.862939 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.863417 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.863421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:05.863516 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:05.863936 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:05.863947 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:05.865090 (+ 1143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:05.866689 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:05.866692 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:05.866694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:05.866884 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:05.867165 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:05.867170 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:05.867175 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:05.867259 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:05.867640 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:05.867643 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:05.867745 (+ 102us) client_negotiation.cc:769] Sending connection context
0923 17:25:05.867851 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 17:25:05.867989 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260923 17:25:05.868201 19020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.860658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52170 (local address 127.11.229.61:41129)
0923 17:25:05.860935 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:05.860938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:05.860949 (+ 11us) server_negotiation.cc:407] Connection header received
0923 17:25:05.860986 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:05.860988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:05.861026 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:05.861126 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:05.861569 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.862249 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.863060 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.863298 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.863405 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:05.865241 (+ 1836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:05.865244 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:05.865246 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:05.865311 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:05.866571 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:05.866984 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:05.866988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:05.866990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:05.867073 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:05.867352 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:05.867356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:05.867358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:05.867541 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:05.867650 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:05.867915 (+ 265us) server_negotiation.cc:299] Negotiation successful
0923 17:25:05.868038 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":94,"threads_started":1}
I20260923 17:25:05.868985 19016 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45961 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:25:05 dist-test-slave-ptl9 krb5kdc[18252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184305, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.60@KRBTEST.COM for oryx/127.11.229.60@KRBTEST.COM
I20260923 17:25:05.876749 19019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.869547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:45961 (local address 127.0.0.1:42786)
0923 17:25:05.869641 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:05.869657 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:05.869735 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:05.869918 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:05.869920 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:05.870040 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:05.870108 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:05.870112 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.871271 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.871273 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:05.871868 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:05.871878 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.872283 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.872286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:05.872343 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:05.872749 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:05.872760 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:05.873959 (+ 1199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:05.875550 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:05.875553 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:05.875554 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:05.875742 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:05.875969 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:05.875971 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:05.875972 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:05.876041 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:05.876353 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:05.876355 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:05.876427 (+ 72us) client_negotiation.cc:769] Sending connection context
0923 17:25:05.876505 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 17:25:05.876601 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260923 17:25:05.876777 19018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.869609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42786 (local address 127.11.229.60:45961)
0923 17:25:05.869770 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:05.869774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:05.869784 (+ 10us) server_negotiation.cc:407] Connection header received
0923 17:25:05.869813 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:05.869815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:05.869848 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:05.869909 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:05.870200 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.871162 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.871966 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.872187 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.872267 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:05.874079 (+ 1812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:05.874082 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:05.874084 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:05.874105 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:05.875462 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:05.875822 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:05.875825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:05.875826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:05.875893 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:05.876119 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:05.876121 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:05.876122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:05.876276 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:05.876348 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:05.876560 (+ 212us) server_negotiation.cc:299] Negotiation successful
0923 17:25:05.876656 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260923 17:25:05.879753 19016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae: Bootstrap starting.
I20260923 17:25:05.882289 19016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:05.883093 19016 log.cc:824] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae: Log is configured to *not* fsync() on all Append() calls
I20260923 17:25:05.885057 19016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae: No bootstrap required, opened a new log
I20260923 17:25:05.888398 19016 raft_consensus.cc:348] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41129 } } peers { permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45961 } }
I20260923 17:25:05.888679 19016 raft_consensus.cc:374] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:05.888773 19016 raft_consensus.cc:729] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b163f7eb24d54419aa3e1337a7c013ae, State: Initialized, Role: FOLLOWER
I20260923 17:25:05.889307 19016 consensus_queue.cc:260] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41129 } } peers { permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45961 } }
I20260923 17:25:05.889818 19022 sys_catalog.cc:455] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41129 } } peers { permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45961 } } }
I20260923 17:25:05.889933 19022 sys_catalog.cc:458] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:25:05.890573 19016 sys_catalog.cc:564] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [sys.catalog]: configured and running, proceeding with master startup.
W20260923 17:25:05.895183 19033 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:25:05.895272 19033 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:25:05.898792 18868 master_runner.cc:428] Detected that this master b163f7eb24d54419aa3e1337a7c013ae is joining an existing cluster
I20260923 17:25:05.898862 18868 master_runner.cc:432] Initiating AddMaster RPC to add 127.11.229.60:45961
I20260923 17:25:05.909914 19018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.901121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42794 (local address 127.11.229.60:45961)
0923 17:25:05.901233 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:05.901238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:05.901853 (+ 615us) server_negotiation.cc:407] Connection header received
0923 17:25:05.902018 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:05.902020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:05.902055 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:05.902103 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:05.902886 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.903586 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.904952 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.905362 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.905478 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:05.906783 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:05.906787 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:05.906789 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:05.906810 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:05.908243 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:05.908693 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:05.908697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:05.908698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:05.908773 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:05.909042 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:05.909048 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:05.909050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:05.909220 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:05.909370 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:05.909618 (+ 248us) server_negotiation.cc:299] Negotiation successful
0923 17:25:05.909763 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":42}
I20260923 17:25:05.909927 19041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.901336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:45961 (local address 127.0.0.1:42794)
0923 17:25:05.901769 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:05.901783 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:05.901922 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:05.902236 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:05.902241 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:05.902582 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:05.902770 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:05.902779 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.903956 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.903959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:05.904842 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:05.904850 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.905487 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.905490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:05.905589 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:05.906144 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:05.906160 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:05.906649 (+ 489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:05.908370 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:05.908374 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:05.908376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:05.908579 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:05.908869 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:05.908872 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:05.908874 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:05.908956 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:05.909359 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:05.909362 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:05.909455 (+ 93us) client_negotiation.cc:769] Sending connection context
0923 17:25:05.909559 (+ 104us) client_negotiation.cc:240] Negotiation successful
0923 17:25:05.909714 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":20,"spinlock_wait_cycles":4224,"thread_start_us":88,"threads_started":1}
I20260923 17:25:05.911377 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 18359
I20260923 17:25:05.919899 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.1:36025
--local_ip_for_outbound_sockets=127.11.229.1
--tserver_master_addrs=127.11.229.62:46815
--webserver_port=40443
--webserver_interface=127.11.229.1
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.1.keytab
--principal=oryx/127.11.229.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:44685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.229.62:46815,127.11.229.61:41129
--tserver_master_addrs=127.11.229.62:46815,127.11.229.61:41129,127.11.229.60:45961 with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:25:05.934384 19020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.901483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52178 (local address 127.11.229.61:41129)
0923 17:25:05.901600 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:05.901606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:05.901763 (+ 157us) server_negotiation.cc:407] Connection header received
0923 17:25:05.901951 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:05.901955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:05.902008 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:05.902060 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:05.902927 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.903773 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.905583 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.905978 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.906067 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:05.930114 (+ 24047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:05.930120 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:05.930124 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:05.930159 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:05.932320 (+ 2161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:05.932842 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:05.932846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:05.932848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:05.932953 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:05.933280 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:05.933284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:05.933286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:05.933545 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:05.933633 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:05.934060 (+ 427us) server_negotiation.cc:299] Negotiation successful
0923 17:25:05.934202 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 17:25:05.934410 19039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.901402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46815 (local address 127.0.0.1:44962)
0923 17:25:05.901785 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:05.901797 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:05.901922 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:05.902282 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:05.902287 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:05.903329 (+ 1042us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:05.903441 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:05.903448 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.904479 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.904482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:05.905661 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:05.905672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.906331 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.906336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:05.906476 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:05.929937 (+ 23461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:05.929960 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:05.930472 (+ 512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:05.932678 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:05.932682 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:05.932684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:05.932943 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:05.933240 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:05.933243 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:05.933244 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:05.933358 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:05.933838 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:05.933841 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:05.933934 (+ 93us) client_negotiation.cc:769] Sending connection context
0923 17:25:05.934019 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 17:25:05.934203 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":24,"spinlock_wait_cycles":7808,"thread_start_us":97,"threads_started":1}
I20260923 17:25:05.935041 18336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.901159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44962 (local address 127.11.229.62:46815)
0923 17:25:05.901300 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:05.901305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:05.901857 (+ 552us) server_negotiation.cc:407] Connection header received
0923 17:25:05.902030 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:05.902033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:05.902082 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:05.902133 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:05.903573 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.904345 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.905798 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.906082 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.906172 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:05.930675 (+ 24503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:05.930680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:05.930683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:05.930717 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:05.932531 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:05.933055 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:05.933058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:05.933060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:05.933143 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:05.933460 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:05.933463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:05.933465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:05.933709 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:05.933796 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:05.934757 (+ 961us) server_negotiation.cc:299] Negotiation successful
0923 17:25:05.934883 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 17:25:05.935765 19040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.901024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:41129 (local address 127.0.0.1:52178)
0923 17:25:05.901680 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:05.901695 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:05.901838 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:05.902213 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:05.902216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:05.902605 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:05.902780 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:05.902789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.903979 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.903983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:05.905439 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:05.905451 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.906232 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.906235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:05.906331 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:05.906840 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:05.906855 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:05.929836 (+ 22981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:05.932469 (+ 2633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:05.932473 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:05.932476 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:05.932727 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:05.933069 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:05.933072 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:05.933074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:05.933159 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:05.933759 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:05.933762 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:05.933865 (+ 103us) client_negotiation.cc:769] Sending connection context
0923 17:25:05.935017 (+ 1152us) client_negotiation.cc:240] Negotiation successful
0923 17:25:05.935454 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":132,"thread_start_us":92,"threads_started":1}
Sep 23 17:25:05 dist-test-slave-ptl9 krb5kdc[18252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184304, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.62@KRBTEST.COM for oryx/127.11.229.60@KRBTEST.COM
I20260923 17:25:05.948645 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.939188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:45961 (local address 127.0.0.1:42798)
0923 17:25:05.939316 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:05.939331 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:05.939416 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:05.939717 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:05.939721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:05.939913 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:05.940054 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:05.940062 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.941098 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.941101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:05.942101 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:05.942114 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.942734 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.942738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:05.942842 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:05.943340 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:05.943355 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:05.945073 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:05.947018 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:05.947021 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:05.947023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:05.947276 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:05.947574 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:05.947577 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:05.947579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:05.947673 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:05.948095 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:05.948097 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:05.948201 (+ 104us) client_negotiation.cc:769] Sending connection context
0923 17:25:05.948312 (+ 111us) client_negotiation.cc:240] Negotiation successful
0923 17:25:05.948461 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260923 17:25:05.948670 19018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.939263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42798 (local address 127.11.229.60:45961)
0923 17:25:05.939377 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:05.939382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:05.939395 (+ 13us) server_negotiation.cc:407] Connection header received
0923 17:25:05.939530 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:05.939533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:05.939596 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:05.939671 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:05.940173 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.940973 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.942224 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.942613 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.942720 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:05.945251 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:05.945254 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:05.945292 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:05.945320 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:05.946894 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:05.947380 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:05.947384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:05.947386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:05.947471 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:05.947767 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:05.947770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:05.947772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:05.947994 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:05.948089 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:05.948380 (+ 291us) server_negotiation.cc:299] Negotiation successful
0923 17:25:05.948501 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 17:25:05.949990 18297 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.11.229.60:45961: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae"
member_type: NON_VOTER
last_known_addr {
host: "127.11.229.60"
port: 45961
}
attrs {
promote: true
}
}
dest_uuid: "b5536110589147d9b37d53ea18ab3f26"
cas_config_opid_index: 5
I20260923 17:25:05.950618 18297 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: NON_VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: true } } peers { permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae" member_type: NON_VOTER last_known_addr { host: "127.11.229.60" port: 45961 } attrs { promote: true } }
I20260923 17:25:05.951709 18858 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b163f7eb24d54419aa3e1337a7c013ae (127.11.229.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: NON_VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: true } } peers { permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae" member_type: NON_VOTER last_known_addr { host: "127.11.229.60" port: 45961 } attrs { promote: true } } }
I20260923 17:25:05.952924 18990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:05.953765 18858 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.11.229.60:45961
I20260923 17:25:05.953981 18858 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b5536110589147d9b37d53ea18ab3f26" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: NON_VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: true } } peers { permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae" member_type: NON_VOTER last_known_addr { host: "127.11.229.60" port: 45961 } attrs { promote: true } } }
I20260923 17:25:05.954082 18858 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [sys.catalog]: This master's current role is: LEADER
I20260923 17:25:05.954484 18868 master.cc:537] Master@127.11.229.60:45961 shutting down...
I20260923 17:25:05.954533 18990 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [term 1 FOLLOWER]: Refusing update from remote peer b5536110589147d9b37d53ea18ab3f26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260923 17:25:05.956389 18857 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae" member_type: NON_VOTER last_known_addr { host: "127.11.229.60" port: 45961 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 17:25:05.958317 18284 consensus_peers.cc:487] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 -> Peer b163f7eb24d54419aa3e1337a7c013ae (127.11.229.60:45961): Couldn't send request to peer b163f7eb24d54419aa3e1337a7c013ae. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260923 17:25:05.958526 19020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.952020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52190 (local address 127.11.229.61:41129)
0923 17:25:05.953407 (+ 1387us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:05.953411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:05.953427 (+ 16us) server_negotiation.cc:407] Connection header received
0923 17:25:05.953480 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:05.953483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:05.953546 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:05.953649 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:05.954268 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.955049 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.956952 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.957622 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.957706 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:05.957862 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:05.958220 (+ 358us) server_negotiation.cc:299] Negotiation successful
0923 17:25:05.958307 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1292}
I20260923 17:25:05.959254 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.951786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:41129 (local address 127.0.0.1:52190)
0923 17:25:05.952629 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:05.952644 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:05.952720 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:05.953694 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:05.953698 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:05.953718 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:05.953812 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:05.953897 (+ 85us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.955450 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.955453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:05.956806 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:05.956819 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:05.957957 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:05.957960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:05.958050 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:05.958101 (+ 51us) client_negotiation.cc:769] Sending connection context
0923 17:25:05.959049 (+ 948us) client_negotiation.cc:240] Negotiation successful
0923 17:25:05.959138 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754}
I20260923 17:25:05.959918 18908 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [term 1 LEARNER]: Refusing update from remote peer b5536110589147d9b37d53ea18ab3f26: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260923 17:25:05.960481 18857 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: NON_VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260923 17:25:05.962594 18908 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b163f7eb24d54419aa3e1337a7c013ae (127.11.229.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: NON_VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: true } } peers { permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae" member_type: NON_VOTER last_known_addr { host: "127.11.229.60" port: 45961 } attrs { promote: true } } }
I20260923 17:25:05.965138 18937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5536110589147d9b37d53ea18ab3f26. Latest consensus state: current_term: 1 leader_uuid: "b5536110589147d9b37d53ea18ab3f26" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: NON_VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: true } } peers { permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae" member_type: NON_VOTER last_known_addr { host: "127.11.229.60" port: 45961 } attrs { promote: true } } }
I20260923 17:25:05.965287 18937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [sys.catalog]: This master's current role is: LEARNER
I20260923 17:25:05.965381 18937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b5536110589147d9b37d53ea18ab3f26" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: NON_VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: true } } peers { permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae" member_type: NON_VOTER last_known_addr { host: "127.11.229.60" port: 45961 } attrs { promote: true } } }
I20260923 17:25:05.965461 18937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [sys.catalog]: This master's current role is: LEARNER
I20260923 17:25:05.966590 18858 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26: attempting to promote NON_VOTER 23b480ef1799433abb57f326cf3078be to VOTER
I20260923 17:25:05.967144 18858 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: false } } peers { permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae" member_type: NON_VOTER last_known_addr { host: "127.11.229.60" port: 45961 } attrs { promote: true } }
I20260923 17:25:05.968529 18908 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [term 1 LEARNER]: Refusing update from remote peer b5536110589147d9b37d53ea18ab3f26: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260923 17:25:05.968957 18858 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260923 17:25:05.969199 18284 consensus_peers.cc:487] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 -> Peer b163f7eb24d54419aa3e1337a7c013ae (127.11.229.60:45961): Couldn't send request to peer b163f7eb24d54419aa3e1337a7c013ae. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260923 17:25:05.969406 18868 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:25:05.969682 18868 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:25:05.969779 18868 tablet_replica.cc:330] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae: stopping tablet replica
I20260923 17:25:05.970955 18857 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 23b480ef1799433abb57f326cf3078be (127.11.229.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: false } } peers { permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae" member_type: NON_VOTER last_known_addr { host: "127.11.229.60" port: 45961 } attrs { promote: true } } }
I20260923 17:25:05.972416 18857 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b5536110589147d9b37d53ea18ab3f26" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: false } } peers { permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae" member_type: NON_VOTER last_known_addr { host: "127.11.229.60" port: 45961 } attrs { promote: true } } }
I20260923 17:25:05.972520 18857 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [sys.catalog]: This master's current role is: LEADER
I20260923 17:25:05.973902 18858 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b5536110589147d9b37d53ea18ab3f26" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: false } } peers { permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae" member_type: NON_VOTER last_known_addr { host: "127.11.229.60" port: 45961 } attrs { promote: true } } }
I20260923 17:25:05.974009 18858 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [sys.catalog]: This master's current role is: LEADER
I20260923 17:25:05.974133 18908 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 23b480ef1799433abb57f326cf3078be (127.11.229.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: false } } peers { permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae" member_type: NON_VOTER last_known_addr { host: "127.11.229.60" port: 45961 } attrs { promote: true } } }
I20260923 17:25:05.976121 19044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b5536110589147d9b37d53ea18ab3f26" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: false } } peers { permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae" member_type: NON_VOTER last_known_addr { host: "127.11.229.60" port: 45961 } attrs { promote: true } } }
I20260923 17:25:05.976246 19044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:25:05.987352 18868 master.cc:552] Master@127.11.229.60:45961 shutdown complete.
I20260923 17:25:05.987422 18868 master_runner.cc:305] Clearing existing system tablet
I20260923 17:25:05.987684 18868 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:05.987744 18868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:05.989344 18868 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 17:25:05.990517 19047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:05.990778 18868 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:05.990913 18868 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 17:25:05.990957 18868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-2/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-2/wal
uuid: "b163f7eb24d54419aa3e1337a7c013ae"
format_stamp: "Formatted at 2026-09-23 17:25:05 on dist-test-slave-ptl9"
server_key: "d8df0e242c0583185e6a84b44d31d071"
server_key_iv: "a74e29289ee9fa4295ea50172a653b60"
server_key_version: "encryptionkey@0"
I20260923 17:25:05.993161 18868 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:25:05.995643 18868 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 17:25:05.995733 18868 log.cc:1196] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260923 17:25:05.995986 18868 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae: Deleting consensus metadata
I20260923 17:25:05.996341 18868 master_runner.cc:315] Copying system tablet from 127.11.229.62:46815
I20260923 17:25:05.997583 18868 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae: tablet copy: Beginning tablet copy session from remote peer at address 127.11.229.62:46815
I20260923 17:25:06.010120 19053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.998149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46815 (local address 127.0.0.1:44970)
0923 17:25:06.001442 (+ 3293us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:06.001458 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:06.001592 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:06.001884 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:06.001887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:06.002122 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:06.002235 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:06.002241 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.003348 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.003352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:06.004272 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:06.004287 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.004914 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.004920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:06.005067 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.005766 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:06.005782 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:06.006223 (+ 441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:06.008411 (+ 2188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:06.008414 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:06.008416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:06.008685 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:06.008993 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:06.008996 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:06.008998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:06.009098 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:06.009564 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:06.009567 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:06.009671 (+ 104us) client_negotiation.cc:769] Sending connection context
0923 17:25:06.009783 (+ 112us) client_negotiation.cc:240] Negotiation successful
0923 17:25:06.009934 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3208,"thread_start_us":83,"threads_started":1}
I20260923 17:25:06.010146 18336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:05.998253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44970 (local address 127.11.229.62:46815)
0923 17:25:05.998381 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:05.998386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:06.001572 (+ 3186us) server_negotiation.cc:407] Connection header received
0923 17:25:06.001686 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:06.001689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:06.001751 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:06.001854 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:06.002370 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.003212 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.004414 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.004758 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.004864 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.006512 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:06.006516 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:06.006519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:06.006549 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:06.008290 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:06.008790 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:06.008794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:06.008796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:06.008887 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:06.009204 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:06.009207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:06.009209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:06.009451 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:06.009552 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:06.009861 (+ 309us) server_negotiation.cc:299] Negotiation successful
0923 17:25:06.009996 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 17:25:06.011091 18317 tablet_copy_service.cc:140] P b5536110589147d9b37d53ea18ab3f26: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b163f7eb24d54419aa3e1337a7c013ae ({username='oryx', principal='oryx/127.11.229.60@KRBTEST.COM'} at 127.0.0.1:44970)
I20260923 17:25:06.011206 18317 tablet_copy_service.cc:161] P b5536110589147d9b37d53ea18ab3f26: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b163f7eb24d54419aa3e1337a7c013ae at {username='oryx', principal='oryx/127.11.229.60@KRBTEST.COM'} at 127.0.0.1:44970: session id = b163f7eb24d54419aa3e1337a7c013ae-00000000000000000000000000000000
I20260923 17:25:06.012362 18317 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 17:25:06.013844 18868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 17:25:06.016381 18868 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae: tablet copy: Starting download of 0 data blocks...
I20260923 17:25:06.016561 18868 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae: tablet copy: Starting download of 1 WAL segments...
I20260923 17:25:06.022320 18868 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 17:25:06.024437 18317 tablet_copy_service.cc:342] P b5536110589147d9b37d53ea18ab3f26: Request end of tablet copy session b163f7eb24d54419aa3e1337a7c013ae-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.11.229.60@KRBTEST.COM'} at 127.0.0.1:44970
I20260923 17:25:06.024530 18317 tablet_copy_service.cc:433] P b5536110589147d9b37d53ea18ab3f26: ending tablet copy session b163f7eb24d54419aa3e1337a7c013ae-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b163f7eb24d54419aa3e1337a7c013ae
I20260923 17:25:06.032828 18868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:06.034736 19057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:06.034765 19054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:06.037442 18868 server_base.cc:1034] running on GCE node
W20260923 17:25:06.037487 19055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:06.037724 18868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:06.038036 18868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:06.039193 18868 hybrid_clock.cc:648] HybridClock initialized: now 1790184306039179 us; error 31 us; skew 500 ppm
I20260923 17:25:06.039775 18622 heartbeater.cc:503] Master 127.11.229.62:46815 was elected leader, sending a full tablet report...
Sep 23 17:25:06 dist-test-slave-ptl9 krb5kdc[18252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184306, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:06.041049 18868 init.cc:377] Logged in from keytab as oryx/127.11.229.60@KRBTEST.COM (short username oryx)
I20260923 17:25:06.041531 18868 webserver.cc:460] Webserver started at http://127.11.229.60:34839/ using document root <none> and password file <none>
I20260923 17:25:06.041738 18868 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:06.041785 18868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:06.043406 18868 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 17:25:06.044634 19064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:06.044927 18868 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 17:25:06.045007 18868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-2/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-2/wal
uuid: "b163f7eb24d54419aa3e1337a7c013ae"
format_stamp: "Formatted at 2026-09-23 17:25:05 on dist-test-slave-ptl9"
server_key: "d8df0e242c0583185e6a84b44d31d071"
server_key_iv: "a74e29289ee9fa4295ea50172a653b60"
server_key_version: "encryptionkey@0"
I20260923 17:25:06.045090 18868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:06.059073 18868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:06.061385 18868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:06.066748 18868 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.60:45961
I20260923 17:25:06.066771 19119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.60:45961 every 8 connection(s)
I20260923 17:25:06.067417 18868 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/master-2/data/info.pb
I20260923 17:25:06.069166 19120 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:25:06.070288 19120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae: Bootstrap starting.
W20260923 17:25:06.070185 19042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:06.070528 19042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:06.070600 19042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:06.076515 19042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:06.076596 19042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:06.076682 19042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.1
I20260923 17:25:06.081450 19120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:25:06.081776 19120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae: Bootstrap complete.
I20260923 17:25:06.082345 19120 raft_consensus.cc:348] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: false } } peers { permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae" member_type: NON_VOTER last_known_addr { host: "127.11.229.60" port: 45961 } attrs { promote: true } }
I20260923 17:25:06.082490 19120 raft_consensus.cc:729] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b163f7eb24d54419aa3e1337a7c013ae, State: Initialized, Role: LEARNER
I20260923 17:25:06.082592 19120 consensus_queue.cc:260] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: false } } peers { permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae" member_type: NON_VOTER last_known_addr { host: "127.11.229.60" port: 45961 } attrs { promote: true } }
I20260923 17:25:06.083053 19120 sys_catalog.cc:564] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:25:06.082981 19123 sys_catalog.cc:455] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: false } } peers { permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae" member_type: NON_VOTER last_known_addr { host: "127.11.229.60" port: 45961 } attrs { promote: true } } }
I20260923 17:25:06.083151 19123 sys_catalog.cc:458] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [sys.catalog]: This master's current role is: LEARNER
I20260923 17:25:06.084640 19042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:44685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.1.keytab
--principal=oryx/127.11.229.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.1:36025
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.229.1
--webserver_port=40443
--tserver_master_addrs=127.11.229.62:46815,127.11.229.61:41129,127.11.229.60:45961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.1
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:25:06.086110 19134 catalog_manager.cc:1259] Loaded cluster ID: 78a04d62a08a462dab6b3c54b4731f38
I20260923 17:25:06.086175 19134 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae: loading cluster ID for follower catalog manager: success
I20260923 17:25:06.086267 19042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:06.087329 19042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 17:25:06.087800 19134 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae: acquiring CA information for follower catalog manager: success
I20260923 17:25:06.088994 19134 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260923 17:25:06.097481 19136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:06.097640 19137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:06.097637 19042 server_base.cc:1034] running on GCE node
W20260923 17:25:06.097457 19139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:06.098163 19042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:06.098872 19042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:06.100068 19042 hybrid_clock.cc:648] HybridClock initialized: now 1790184306100027 us; error 55 us; skew 500 ppm
Sep 23 17:25:06 dist-test-slave-ptl9 krb5kdc[18252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184306, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:06.102771 19042 init.cc:377] Logged in from keytab as oryx/127.11.229.1@KRBTEST.COM (short username oryx)
I20260923 17:25:06.103612 19042 webserver.cc:460] Webserver started at http://127.11.229.1:40443/ using document root <none> and password file <none>
I20260923 17:25:06.104178 19042 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:06.104246 19042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:06.108109 19042 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:25:06.110240 19146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:06.111634 19042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:25:06.111749 19042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-0/wal
uuid: "e507acf6b90b45269c0f77c976e485ca"
format_stamp: "Formatted at 2026-09-23 17:25:04 on dist-test-slave-ptl9"
server_key: "74a9f52ebb4b7b1ec000a49416ece6ca"
server_key_iv: "f2d9e102488f71dda0b9e88c3a9c7091"
server_key_version: "encryptionkey@0"
I20260923 17:25:06.112304 19042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:06.156493 19042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:06.159544 19042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:06.159718 19042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:06.160291 19042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:25:06.161159 19042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:25:06.161206 19042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:06.161248 19042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:25:06.161337 19042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:06.170413 19042 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.1:36025
I20260923 17:25:06.170535 19259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.1:36025 every 8 connection(s)
I20260923 17:25:06.171622 19042 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
I20260923 17:25:06.175470 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 19042
I20260923 17:25:06.175684 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 18493
I20260923 17:25:06.184317 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.2:36405
--local_ip_for_outbound_sockets=127.11.229.2
--tserver_master_addrs=127.11.229.62:46815
--webserver_port=46283
--webserver_interface=127.11.229.2
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.2.keytab
--principal=oryx/127.11.229.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:44685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.229.62:46815,127.11.229.61:41129
--tserver_master_addrs=127.11.229.62:46815,127.11.229.61:41129,127.11.229.60:45961 with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:25:06 dist-test-slave-ptl9 krb5kdc[18252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184306, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.60@KRBTEST.COM
Sep 23 17:25:06 dist-test-slave-ptl9 krb5kdc[18252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184306, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM
Sep 23 17:25:06 dist-test-slave-ptl9 krb5kdc[18252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184306, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.61@KRBTEST.COM
I20260923 17:25:06.190830 19020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.174284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:53579 (local address 127.11.229.61:41129)
0923 17:25:06.174433 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:06.174439 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:06.176197 (+ 1758us) server_negotiation.cc:407] Connection header received
0923 17:25:06.176260 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:06.176266 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:06.176335 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:06.176419 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:06.178675 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.179579 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.182681 (+ 3102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.183050 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.183168 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.186217 (+ 3049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:06.186221 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:06.186225 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:06.186255 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:06.188063 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:06.188842 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:06.188846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:06.188848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:06.188952 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:06.189416 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:06.189419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:06.189421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:06.189647 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:06.189737 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:06.190517 (+ 780us) server_negotiation.cc:299] Negotiation successful
0923 17:25:06.190668 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 17:25:06.191059 18336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.175139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:49975 (local address 127.11.229.62:46815)
0923 17:25:06.175509 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:06.175514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:06.175533 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:25:06.176489 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:06.176493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:06.176548 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:06.176606 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:06.178516 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.179334 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.181444 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.181798 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.181880 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.186082 (+ 4202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:06.186087 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:06.186092 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:06.186127 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:06.188747 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:06.189414 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:06.189420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:06.189423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:06.189533 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:06.189886 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:06.189892 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:06.189897 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:06.190158 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:06.190257 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:06.190680 (+ 423us) server_negotiation.cc:299] Negotiation successful
0923 17:25:06.190825 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278}
I20260923 17:25:06.192350 19267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.175222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:39691 (local address 127.11.229.60:45961)
0923 17:25:06.176204 (+ 982us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:06.176210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:06.176228 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:25:06.177804 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:06.177807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:06.177861 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:06.177934 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:06.178506 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.179578 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.181443 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.181798 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.181890 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.186065 (+ 4175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:06.186070 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:06.186075 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:06.186110 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:06.188001 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:06.188842 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:06.188846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:06.188848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:06.188952 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:06.189416 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:06.189419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:06.189421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:06.189647 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:06.189740 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:06.192051 (+ 2311us) server_negotiation.cc:299] Negotiation successful
0923 17:25:06.192187 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":889,"thread_start_us":88,"threads_started":1}
I20260923 17:25:06.192447 19264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.174077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46815 (local address 127.11.229.1:49975)
0923 17:25:06.175270 (+ 1193us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:06.175285 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:06.175798 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:06.176873 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:06.176880 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:06.177326 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:06.178337 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:06.178357 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.179504 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.179507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:06.181301 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:06.181313 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.182063 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.182067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:06.182379 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.183661 (+ 1282us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:06.183676 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:06.185833 (+ 2157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:06.188898 (+ 3065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:06.188901 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:06.188903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:06.189192 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:06.189663 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:06.189668 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:06.189672 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:06.189771 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:06.190325 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:06.190331 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:06.190443 (+ 112us) client_negotiation.cc:769] Sending connection context
0923 17:25:06.191285 (+ 842us) client_negotiation.cc:240] Negotiation successful
0923 17:25:06.191468 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1097,"mutex_wait_us":151,"spinlock_wait_cycles":389632,"thread_start_us":112,"threads_started":1}
I20260923 17:25:06.192725 19265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.174077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:41129 (local address 127.11.229.1:53579)
0923 17:25:06.175735 (+ 1658us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:06.175750 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:06.175848 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:06.177161 (+ 1313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:06.177167 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:06.177448 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:06.178496 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:06.178526 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.179732 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.179738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:06.182556 (+ 2818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:06.182571 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.183181 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.183188 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:06.183307 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.184061 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:06.184080 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:06.186054 (+ 1974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:06.188182 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:06.188189 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:06.188208 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:06.188722 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:06.189061 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:06.189063 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:06.189065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:06.189311 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:06.189797 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:06.189804 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:06.190222 (+ 418us) client_negotiation.cc:769] Sending connection context
0923 17:25:06.191057 (+ 835us) client_negotiation.cc:240] Negotiation successful
0923 17:25:06.191271 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1568,"mutex_wait_us":1042,"spinlock_wait_cycles":29824,"thread_start_us":107,"threads_started":1}
I20260923 17:25:06.193547 19266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.174076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:45961 (local address 127.11.229.1:39691)
0923 17:25:06.174683 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:06.174737 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:06.177637 (+ 2900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:06.178058 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:06.178061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:06.178246 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:06.178340 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:06.178357 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.179696 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.179700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:06.181301 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:06.181313 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.182063 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.182067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:06.182379 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.183209 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:06.183232 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:06.185877 (+ 2645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:06.188182 (+ 2305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:06.188189 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:06.188208 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:06.188722 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:06.189060 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:06.189063 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:06.189065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:06.189299 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:06.189797 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:06.189804 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:06.190222 (+ 418us) client_negotiation.cc:769] Sending connection context
0923 17:25:06.190628 (+ 406us) client_negotiation.cc:240] Negotiation successful
0923 17:25:06.193354 (+ 2726us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":656,"spinlock_wait_cycles":15232,"thread_start_us":143,"threads_started":1}
I20260923 17:25:06.194068 19262 heartbeater.cc:347] Connected to a master server at 127.11.229.61:41129
I20260923 17:25:06.194341 19262 heartbeater.cc:464] Registering TS with master...
I20260923 17:25:06.194528 19260 heartbeater.cc:347] Connected to a master server at 127.11.229.60:45961
I20260923 17:25:06.194609 19260 heartbeater.cc:464] Registering TS with master...
I20260923 17:25:06.194873 19261 heartbeater.cc:347] Connected to a master server at 127.11.229.62:46815
I20260923 17:25:06.194947 19261 heartbeater.cc:464] Registering TS with master...
I20260923 17:25:06.195040 19262 heartbeater.cc:511] Master 127.11.229.61:41129 requested a full tablet report, sending...
I20260923 17:25:06.195040 19260 heartbeater.cc:511] Master 127.11.229.60:45961 requested a full tablet report, sending...
I20260923 17:25:06.195113 19261 heartbeater.cc:511] Master 127.11.229.62:46815 requested a full tablet report, sending...
I20260923 17:25:06.196498 18297 ts_manager.cc:194] Re-registered known tserver with Master: e507acf6b90b45269c0f77c976e485ca (127.11.229.1:36025)
I20260923 17:25:06.196697 18898 ts_manager.cc:194] Registered new tserver with Master: e507acf6b90b45269c0f77c976e485ca (127.11.229.1:36025)
I20260923 17:25:06.197338 18297 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.229.1@KRBTEST.COM'} at 127.11.229.1:49975
I20260923 17:25:06.197705 19084 ts_manager.cc:194] Registered new tserver with Master: e507acf6b90b45269c0f77c976e485ca (127.11.229.1:36025)
I20260923 17:25:06.275435 18756 heartbeater.cc:503] Master 127.11.229.62:46815 was elected leader, sending a full tablet report...
W20260923 17:25:06.303078 19268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:06.303366 19268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:06.303457 19268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:06.307802 19268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:06.307878 19268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:06.307969 19268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.2
I20260923 17:25:06.313652 19268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:44685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.2.keytab
--principal=oryx/127.11.229.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.2:36405
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.229.2
--webserver_port=46283
--tserver_master_addrs=127.11.229.62:46815,127.11.229.61:41129,127.11.229.60:45961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.2
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:25:06.314894 19268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:06.316087 19268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:06.325902 19276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:06.325958 19274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:06.326025 19273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:06.325944 19268 server_base.cc:1034] running on GCE node
I20260923 17:25:06.326615 19268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:06.327358 19268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:06.328581 19268 hybrid_clock.cc:648] HybridClock initialized: now 1790184306328560 us; error 40 us; skew 500 ppm
Sep 23 17:25:06 dist-test-slave-ptl9 krb5kdc[18252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184306, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:06.331521 19268 init.cc:377] Logged in from keytab as oryx/127.11.229.2@KRBTEST.COM (short username oryx)
I20260923 17:25:06.332377 19268 webserver.cc:460] Webserver started at http://127.11.229.2:46283/ using document root <none> and password file <none>
I20260923 17:25:06.332955 19268 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:06.333029 19268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:06.336985 19268 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:25:06.339208 19283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:06.340528 19268 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260923 17:25:06.340698 19268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-1/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-1/wal
uuid: "50709fadaa9e44cf830396ddbb1b72a4"
format_stamp: "Formatted at 2026-09-23 17:25:04 on dist-test-slave-ptl9"
server_key: "6fb8dd01dbf121364078ea16feba0206"
server_key_iv: "35405b5b7c85db3d61328b038dcd9f66"
server_key_version: "encryptionkey@0"
I20260923 17:25:06.341321 19268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:06.354377 19268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:06.357604 19268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:06.357824 19268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:06.358453 19268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:25:06.359357 19268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:25:06.359432 19268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:06.359520 19268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:25:06.359565 19268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:06.369441 19268 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.2:36405
I20260923 17:25:06.369513 19396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.2:36405 every 8 connection(s)
I20260923 17:25:06.370671 19268 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
I20260923 17:25:06.371879 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 19268
I20260923 17:25:06.372048 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 18627
I20260923 17:25:06.380697 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.3:35491
--local_ip_for_outbound_sockets=127.11.229.3
--tserver_master_addrs=127.11.229.62:46815
--webserver_port=33833
--webserver_interface=127.11.229.3
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.3.keytab
--principal=oryx/127.11.229.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:44685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.229.62:46815,127.11.229.61:41129
--tserver_master_addrs=127.11.229.62:46815,127.11.229.61:41129,127.11.229.60:45961 with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:25:06 dist-test-slave-ptl9 krb5kdc[18252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184306, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM
Sep 23 17:25:06 dist-test-slave-ptl9 krb5kdc[18252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184306, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.61@KRBTEST.COM
Sep 23 17:25:06 dist-test-slave-ptl9 krb5kdc[18252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184306, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.60@KRBTEST.COM
I20260923 17:25:06.390792 19267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.377529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:44417 (local address 127.11.229.60:45961)
0923 17:25:06.377850 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:06.377854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:06.377935 (+ 81us) server_negotiation.cc:407] Connection header received
0923 17:25:06.378110 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:06.378115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:06.378173 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:06.378230 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:06.378922 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.379793 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.381321 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.381749 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.381845 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.386102 (+ 4257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:06.386106 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:06.386109 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:06.386137 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:06.387994 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:06.388708 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:06.388714 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:06.388717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:06.388818 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:06.389308 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:06.389311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:06.389313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:06.389568 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:06.389640 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:06.390411 (+ 771us) server_negotiation.cc:299] Negotiation successful
0923 17:25:06.390555 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243}
I20260923 17:25:06.391395 19020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.375071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:44015 (local address 127.11.229.61:41129)
0923 17:25:06.375204 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:06.375210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:06.375231 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:25:06.376165 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:06.376169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:06.376219 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:06.376332 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:06.378230 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.379119 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.380978 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.381343 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.381420 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.386102 (+ 4682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:06.386106 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:06.386109 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:06.386137 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:06.389480 (+ 3343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:06.389959 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:06.389962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:06.389964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:06.390052 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:06.390361 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:06.390364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:06.390366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:06.390582 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:06.390657 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:06.391096 (+ 439us) server_negotiation.cc:299] Negotiation successful
0923 17:25:06.391253 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 17:25:06.391831 18336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.373240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:49087 (local address 127.11.229.62:46815)
0923 17:25:06.373411 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:06.373416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:06.376453 (+ 3037us) server_negotiation.cc:407] Connection header received
0923 17:25:06.376665 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:06.376669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:06.376733 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:06.376824 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:06.379169 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.379958 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.381125 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.381492 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.381568 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.386199 (+ 4631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:06.386206 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:06.386210 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:06.386249 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:06.388554 (+ 2305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:06.389438 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:06.389442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:06.389444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:06.389651 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:06.390121 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:06.390125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:06.390127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:06.390527 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:06.390644 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:06.391397 (+ 753us) server_negotiation.cc:299] Negotiation successful
0923 17:25:06.391565 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260923 17:25:06.392885 19403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.377090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:45961 (local address 127.11.229.2:44417)
0923 17:25:06.377849 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:06.377865 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:06.378002 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:06.378387 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:06.378392 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:06.378620 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:06.378777 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:06.378787 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.379937 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.379941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:06.381147 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:06.381162 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.382061 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.382066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:06.382213 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.382889 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:06.382908 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:06.385886 (+ 2978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:06.388137 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:06.388144 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:06.388165 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:06.388560 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:06.388952 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:06.388955 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:06.388958 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:06.389151 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:06.389736 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:06.389743 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:06.390107 (+ 364us) client_negotiation.cc:769] Sending connection context
0923 17:25:06.391181 (+ 1074us) client_negotiation.cc:240] Negotiation successful
0923 17:25:06.391700 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"mutex_wait_us":170,"thread_start_us":106,"threads_started":1}
I20260923 17:25:06.392900 19402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.373797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46815 (local address 127.11.229.2:49087)
0923 17:25:06.376367 (+ 2570us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:06.376382 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:06.376544 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:06.376969 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:06.376972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:06.377220 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:06.378995 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:06.379008 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.380107 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.380110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:06.381002 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:06.381011 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.381713 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.381716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:06.381907 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.383046 (+ 1139us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:06.383064 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:06.386008 (+ 2944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:06.388729 (+ 2721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:06.388732 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:06.388734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:06.389125 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:06.389836 (+ 711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:06.389841 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:06.389843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:06.389983 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:06.390733 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:06.390736 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:06.390921 (+ 185us) client_negotiation.cc:769] Sending connection context
0923 17:25:06.392242 (+ 1321us) client_negotiation.cc:240] Negotiation successful
0923 17:25:06.392423 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2451,"mutex_wait_us":162,"spinlock_wait_cycles":61568,"thread_start_us":152,"threads_started":1}
I20260923 17:25:06.393208 19401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.373830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:41129 (local address 127.11.229.2:44015)
0923 17:25:06.374578 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:06.374637 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:06.375911 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:06.376537 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:06.376545 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:06.377220 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:06.378039 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:06.378061 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.379291 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.379294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:06.380684 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:06.380695 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.381631 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.381636 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:06.381907 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.382787 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:06.382813 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:06.385886 (+ 3073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:06.389617 (+ 3731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:06.389619 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:06.389621 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:06.389849 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:06.390175 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:06.390178 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:06.390180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:06.390257 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:06.390733 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:06.390735 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:06.390869 (+ 134us) client_negotiation.cc:769] Sending connection context
0923 17:25:06.391620 (+ 751us) client_negotiation.cc:240] Negotiation successful
0923 17:25:06.391741 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":603,"spinlock_wait_cycles":2048,"thread_start_us":723,"threads_started":1}
I20260923 17:25:06.394740 19399 heartbeater.cc:347] Connected to a master server at 127.11.229.61:41129
I20260923 17:25:06.394733 19397 heartbeater.cc:347] Connected to a master server at 127.11.229.60:45961
I20260923 17:25:06.394733 19398 heartbeater.cc:347] Connected to a master server at 127.11.229.62:46815
I20260923 17:25:06.395021 19399 heartbeater.cc:464] Registering TS with master...
I20260923 17:25:06.395020 19398 heartbeater.cc:464] Registering TS with master...
I20260923 17:25:06.395020 19397 heartbeater.cc:464] Registering TS with master...
I20260923 17:25:06.395678 19397 heartbeater.cc:511] Master 127.11.229.60:45961 requested a full tablet report, sending...
I20260923 17:25:06.395896 19398 heartbeater.cc:511] Master 127.11.229.62:46815 requested a full tablet report, sending...
I20260923 17:25:06.395680 19399 heartbeater.cc:511] Master 127.11.229.61:41129 requested a full tablet report, sending...
I20260923 17:25:06.396953 19084 ts_manager.cc:194] Registered new tserver with Master: 50709fadaa9e44cf830396ddbb1b72a4 (127.11.229.2:36405)
I20260923 17:25:06.397153 18898 ts_manager.cc:194] Registered new tserver with Master: 50709fadaa9e44cf830396ddbb1b72a4 (127.11.229.2:36405)
I20260923 17:25:06.397495 18297 ts_manager.cc:194] Re-registered known tserver with Master: 50709fadaa9e44cf830396ddbb1b72a4 (127.11.229.2:36405)
I20260923 17:25:06.398100 18297 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.229.2@KRBTEST.COM'} at 127.11.229.2:49087
I20260923 17:25:06.428213 19267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.424568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42802 (local address 127.11.229.60:45961)
0923 17:25:06.424717 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:06.424721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:06.424736 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:25:06.424837 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:06.424839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:06.424893 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:06.424974 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:06.425295 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.425818 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.427286 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.427708 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.427792 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.427902 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:06.428046 (+ 144us) server_negotiation.cc:299] Negotiation successful
0923 17:25:06.428106 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:25:06.428234 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.424462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:45961 (local address 127.0.0.1:42802)
0923 17:25:06.424604 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:06.424620 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:06.424728 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:06.425013 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:06.425016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:06.425037 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:06.425156 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:06.425163 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.425929 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.425933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:06.427157 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:06.427169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.427826 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.427830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:06.427917 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.427943 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 17:25:06.428021 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 17:25:06.428095 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260923 17:25:06.428874 19094 raft_consensus.cc:1203] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260923 17:25:06.429298 19123 sys_catalog.cc:455] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5536110589147d9b37d53ea18ab3f26. Latest consensus state: current_term: 1 leader_uuid: "b5536110589147d9b37d53ea18ab3f26" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: false } } peers { permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae" member_type: NON_VOTER last_known_addr { host: "127.11.229.60" port: 45961 } attrs { promote: true } } }
I20260923 17:25:06.429494 19123 sys_catalog.cc:458] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [sys.catalog]: This master's current role is: LEARNER
W20260923 17:25:06.504261 19404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:06.504527 19404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:06.504609 19404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:06.508642 19404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:06.508701 19404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:06.508798 19404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.3
I20260923 17:25:06.514213 19404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:44685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.3.keytab
--principal=oryx/127.11.229.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.3:35491
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.229.3
--webserver_port=33833
--tserver_master_addrs=127.11.229.62:46815,127.11.229.61:41129,127.11.229.60:45961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.3
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:25:06.515281 19404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:06.516315 19404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:06.526482 19410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:06.526503 19409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:06.526482 19412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:06.527302 19404 server_base.cc:1034] running on GCE node
I20260923 17:25:06.527704 19404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:06.528313 19404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:06.529474 19404 hybrid_clock.cc:648] HybridClock initialized: now 1790184306529442 us; error 39 us; skew 500 ppm
Sep 23 17:25:06 dist-test-slave-ptl9 krb5kdc[18252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184306, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:06.532259 19404 init.cc:377] Logged in from keytab as oryx/127.11.229.3@KRBTEST.COM (short username oryx)
I20260923 17:25:06.533200 19404 webserver.cc:460] Webserver started at http://127.11.229.3:33833/ using document root <none> and password file <none>
I20260923 17:25:06.533824 19404 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:06.533917 19404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:06.537761 19404 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260923 17:25:06.539783 19419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:06.540980 19404 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:25:06.541113 19404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-2/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-2/wal
uuid: "59e579024b004c1ebea39d11c0ce6461"
format_stamp: "Formatted at 2026-09-23 17:25:05 on dist-test-slave-ptl9"
server_key: "5ec131cf45ea2f9fd6129d62700997ff"
server_key_iv: "f5db46fdb2bacbfca77a5401d0ac3452"
server_key_version: "encryptionkey@0"
I20260923 17:25:06.541713 19404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:06.560928 19404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:06.563817 19404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:06.564033 19404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:06.564636 19404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:25:06.565580 19404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:25:06.565649 19404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:06.565721 19404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:25:06.565762 19404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:06.575201 19404 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.3:35491
I20260923 17:25:06.575222 19532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.3:35491 every 8 connection(s)
I20260923 17:25:06.576205 19404 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
I20260923 17:25:06.578745 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 19404
Sep 23 17:25:06 dist-test-slave-ptl9 krb5kdc[18252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184306, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.3@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM
Sep 23 17:25:06 dist-test-slave-ptl9 krb5kdc[18252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184306, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.3@KRBTEST.COM for oryx/127.11.229.61@KRBTEST.COM
I20260923 17:25:06.592433 19545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.581388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44986 (local address 127.11.229.62:46815)
0923 17:25:06.581614 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:06.581618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:06.581630 (+ 12us) server_negotiation.cc:407] Connection header received
0923 17:25:06.581669 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:06.581672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:06.581713 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:06.582732 (+ 1019us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:06.582832 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.583656 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.584875 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.585230 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.585733 (+ 503us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.587064 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:06.587068 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:06.587071 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:06.587096 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:06.589135 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:06.589920 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:06.589923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:06.589924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:06.590029 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:06.590362 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:06.590365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:06.590367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:06.590585 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:06.592012 (+ 1427us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:06.592089 (+ 77us) server_negotiation.cc:299] Negotiation successful
0923 17:25:06.592219 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":154,"thread_start_us":72,"threads_started":1}
Sep 23 17:25:06 dist-test-slave-ptl9 krb5kdc[18252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184306, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.3@KRBTEST.COM for oryx/127.11.229.60@KRBTEST.COM
I20260923 17:25:06.595450 19020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.580611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:49537 (local address 127.11.229.61:41129)
0923 17:25:06.580909 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:06.580914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:06.580929 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:25:06.580976 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:06.580982 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:06.581037 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:06.581137 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:06.584337 (+ 3200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.585069 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.586848 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.587177 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.587246 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.590712 (+ 3466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:06.590717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:06.590720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:06.590753 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:06.592967 (+ 2214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:06.593656 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:06.593659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:06.593661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:06.593741 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:06.594172 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:06.594175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:06.594177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:06.594400 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:06.594481 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:06.595194 (+ 713us) server_negotiation.cc:299] Negotiation successful
0923 17:25:06.595309 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208}
I20260923 17:25:06.597200 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.579625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:41129 (local address 127.11.229.3:49537)
0923 17:25:06.580339 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:06.580353 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:06.580437 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:06.581703 (+ 1266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:06.581706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:06.581919 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:06.584175 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:06.584183 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.585223 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.585225 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:06.586475 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:06.586486 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.587391 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.587394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:06.587494 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.587939 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:06.587954 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:06.590147 (+ 2193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:06.593100 (+ 2953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:06.593108 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:06.593128 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:06.593528 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:06.593856 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:06.593860 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:06.593864 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:06.594053 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:06.594548 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:06.594556 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:06.594880 (+ 324us) client_negotiation.cc:769] Sending connection context
0923 17:25:06.595606 (+ 726us) client_negotiation.cc:240] Negotiation successful
0923 17:25:06.596019 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":6,"spinlock_wait_cycles":13696,"thread_start_us":96,"threads_started":1}
I20260923 17:25:06.597904 19542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.578520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46815 (local address 127.11.229.3:46593)
0923 17:25:06.579994 (+ 1474us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:06.580008 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:06.580358 (+ 350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:06.580995 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:06.581002 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:06.581853 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:06.582605 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:06.582625 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.583908 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.583914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:06.585313 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:06.585325 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.586000 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.586005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:06.586232 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.587432 (+ 1200us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:06.587476 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:06.590552 (+ 3076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:06.596325 (+ 5773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:06.596327 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:06.596329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:06.596569 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:06.596873 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:06.596876 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:06.596877 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:06.596957 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:06.597510 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:06.597513 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:06.597605 (+ 92us) client_negotiation.cc:769] Sending connection context
0923 17:25:06.597683 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 17:25:06.597780 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1391,"mutex_wait_us":84,"thread_start_us":123,"threads_started":1}
I20260923 17:25:06.598063 18336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.578232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:46593 (local address 127.11.229.62:46815)
0923 17:25:06.578372 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:06.578377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:06.580108 (+ 1731us) server_negotiation.cc:407] Connection header received
0923 17:25:06.580655 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:06.580658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:06.580722 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:06.580815 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:06.582827 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.583726 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.585447 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.585764 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.585835 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.591111 (+ 5276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:06.591117 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:06.591123 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:06.591155 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:06.595265 (+ 4110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:06.596683 (+ 1418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:06.596687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:06.596689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:06.596770 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:06.597058 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:06.597061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:06.597063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:06.597322 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:06.597421 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:06.597796 (+ 375us) server_negotiation.cc:299] Negotiation successful
0923 17:25:06.597913 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":55}
I20260923 17:25:06.599103 19534 heartbeater.cc:347] Connected to a master server at 127.11.229.62:46815
I20260923 17:25:06.599340 19534 heartbeater.cc:464] Registering TS with master...
I20260923 17:25:06.599856 19534 heartbeater.cc:511] Master 127.11.229.62:46815 requested a full tablet report, sending...
I20260923 17:25:06.597157 18297 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44986:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:25:06.600941 18297 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:25:06.601058 18296 ts_manager.cc:194] Re-registered known tserver with Master: 59e579024b004c1ebea39d11c0ce6461 (127.11.229.3:35491)
I20260923 17:25:06.601728 19535 heartbeater.cc:347] Connected to a master server at 127.11.229.61:41129
I20260923 17:25:06.601824 19535 heartbeater.cc:464] Registering TS with master...
I20260923 17:25:06.601840 18296 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.229.3@KRBTEST.COM'} at 127.11.229.3:46593
I20260923 17:25:06.602010 19535 heartbeater.cc:511] Master 127.11.229.61:41129 requested a full tablet report, sending...
I20260923 17:25:06.602221 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.578519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:45961 (local address 127.11.229.3:50065)
0923 17:25:06.579116 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:06.579181 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:06.580339 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:06.581609 (+ 1270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:06.581612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:06.582450 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:06.582604 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:06.582624 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.583871 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.583874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:06.587248 (+ 3374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:06.587259 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.592063 (+ 4804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.592067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:06.592158 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.592626 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:06.592640 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:06.594283 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:06.598411 (+ 4128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:06.598417 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:06.598418 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:06.600640 (+ 2222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:06.600979 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:06.600982 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:06.600984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:06.601075 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:06.601577 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:06.601579 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:06.601658 (+ 79us) client_negotiation.cc:769] Sending connection context
0923 17:25:06.601895 (+ 237us) client_negotiation.cc:240] Negotiation successful
0923 17:25:06.602046 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":2,"thread_start_us":689,"threads_started":1}
I20260923 17:25:06.602840 18898 ts_manager.cc:194] Registered new tserver with Master: 59e579024b004c1ebea39d11c0ce6461 (127.11.229.3:35491)
I20260923 17:25:06.602870 19267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.578095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:50065 (local address 127.11.229.60:45961)
0923 17:25:06.578224 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:06.578229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:06.579305 (+ 1076us) server_negotiation.cc:407] Connection header received
0923 17:25:06.580579 (+ 1274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:06.580583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:06.580645 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:06.580715 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:06.582765 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.583647 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.591145 (+ 7498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.591508 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.591579 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.594689 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:06.594693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:06.594695 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:06.594719 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:06.596981 (+ 2262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:06.600730 (+ 3749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:06.600733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:06.600735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:06.600854 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:06.601191 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:06.601195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:06.601196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:06.601431 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:06.601518 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:06.601822 (+ 304us) server_negotiation.cc:299] Negotiation successful
0923 17:25:06.602692 (+ 870us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":753,"server-negotiator.queue_time_us":51}
I20260923 17:25:06.603140 19548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.581951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42806 (local address 127.11.229.60:45961)
0923 17:25:06.584729 (+ 2778us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:06.584741 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:06.584760 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:25:06.584793 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:06.584796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:06.585733 (+ 937us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:06.585819 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:06.586699 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.587472 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.588700 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.589037 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.589131 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.590722 (+ 1591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:06.590725 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:06.590728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:06.590754 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:06.594446 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:06.594974 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:06.594982 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:06.594984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:06.595075 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:06.595419 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:06.595423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:06.595424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:06.595656 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:06.595793 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:06.596278 (+ 485us) server_negotiation.cc:299] Negotiation successful
0923 17:25:06.602999 (+ 6721us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2707,"thread_start_us":85,"threads_started":1}
I20260923 17:25:06.603438 19533 heartbeater.cc:347] Connected to a master server at 127.11.229.60:45961
I20260923 17:25:06.603529 19533 heartbeater.cc:464] Registering TS with master...
I20260923 17:25:06.603735 19533 heartbeater.cc:511] Master 127.11.229.60:45961 requested a full tablet report, sending...
I20260923 17:25:06.604374 19084 ts_manager.cc:194] Registered new tserver with Master: 59e579024b004c1ebea39d11c0ce6461 (127.11.229.3:35491)
I20260923 17:25:06.605872 18858 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b5536110589147d9b37d53ea18ab3f26" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: false } } peers { permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae" member_type: NON_VOTER last_known_addr { host: "127.11.229.60" port: 45961 } attrs { promote: true } } }
I20260923 17:25:06.606022 18858 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [sys.catalog]: This master's current role is: LEADER
I20260923 17:25:06.606673 19546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.581416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52194 (local address 127.11.229.61:41129)
0923 17:25:06.582873 (+ 1457us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:06.582879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:06.582891 (+ 12us) server_negotiation.cc:407] Connection header received
0923 17:25:06.582929 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:06.582932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:06.582975 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:06.584304 (+ 1329us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:06.584414 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.585601 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.589797 (+ 4196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.590214 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.590678 (+ 464us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.591938 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:06.591944 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:06.591952 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:06.591984 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:06.604394 (+ 12410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:06.604917 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:06.604921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:06.604923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:06.605018 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:06.605326 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:06.605330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:06.605331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:06.605571 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:06.606312 (+ 741us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:06.606391 (+ 79us) server_negotiation.cc:299] Negotiation successful
0923 17:25:06.606518 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":1381,"thread_start_us":80,"threads_started":1}
I20260923 17:25:06.609710 19551 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26: attempting to promote NON_VOTER b163f7eb24d54419aa3e1337a7c013ae to VOTER
I20260923 17:25:06.610080 19551 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: false } } peers { permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45961 } attrs { promote: false } }
I20260923 17:25:06.611155 19094 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [term 1 LEARNER]: Refusing update from remote peer b5536110589147d9b37d53ea18ab3f26: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260923 17:25:06.611509 18858 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45961 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260923 17:25:06.612751 18908 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [term 1 FOLLOWER]: Refusing update from remote peer b5536110589147d9b37d53ea18ab3f26: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260923 17:25:06.613233 18858 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260923 17:25:06.613579 19551 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b163f7eb24d54419aa3e1337a7c013ae (127.11.229.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: false } } peers { permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45961 } attrs { promote: false } } }
I20260923 17:25:06.614148 19094 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b163f7eb24d54419aa3e1337a7c013ae (127.11.229.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: false } } peers { permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45961 } attrs { promote: false } } }
I20260923 17:25:06.615208 19551 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b5536110589147d9b37d53ea18ab3f26" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: false } } peers { permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45961 } attrs { promote: false } } }
I20260923 17:25:06.615315 19551 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [sys.catalog]: This master's current role is: LEADER
I20260923 17:25:06.615245 19552 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b5536110589147d9b37d53ea18ab3f26" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: false } } peers { permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45961 } attrs { promote: false } } }
I20260923 17:25:06.615363 19552 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [sys.catalog]: This master's current role is: LEADER
I20260923 17:25:06.615541 19551 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b5536110589147d9b37d53ea18ab3f26" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: false } } peers { permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45961 } attrs { promote: false } } }
I20260923 17:25:06.615633 19551 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26 [sys.catalog]: This master's current role is: LEADER
I20260923 17:25:06.615854 18908 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b163f7eb24d54419aa3e1337a7c013ae (127.11.229.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: false } } peers { permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45961 } attrs { promote: false } } }
I20260923 17:25:06.616310 19123 sys_catalog.cc:455] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b5536110589147d9b37d53ea18ab3f26" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: false } } peers { permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45961 } attrs { promote: false } } }
I20260923 17:25:06.616415 19123 sys_catalog.cc:458] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:25:06.618703 19560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b5536110589147d9b37d53ea18ab3f26" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: false } } peers { permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45961 } attrs { promote: false } } }
I20260923 17:25:06.618837 19560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:25:06.624923 19562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.620112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:36025 (local address 127.0.0.1:41218)
0923 17:25:06.620366 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:06.620380 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:06.620460 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:06.621314 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:06.621319 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:06.621337 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:06.621446 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:06.621454 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.622795 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.622800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:06.623556 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:06.623564 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.624437 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.624441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:06.624594 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.624617 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 17:25:06.624693 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 17:25:06.624773 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":24,"thread_start_us":106,"threads_started":1}
I20260923 17:25:06.625535 19564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.620518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41218 (local address 127.11.229.1:36025)
0923 17:25:06.620812 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:06.620841 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:06.620864 (+ 23us) server_negotiation.cc:407] Connection header received
0923 17:25:06.620944 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:06.620952 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:06.621151 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:06.621349 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:06.621549 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.622687 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.623680 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.624308 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.624434 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.624618 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:06.625218 (+ 600us) server_negotiation.cc:299] Negotiation successful
0923 17:25:06.625378 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":87,"threads_started":1}
I20260923 17:25:06.625691 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.619586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:35491 (local address 127.0.0.1:38636)
0923 17:25:06.619712 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:06.619729 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:06.619809 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:06.620737 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:06.620740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:06.620759 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:06.620874 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:06.620883 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.622460 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.622463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:06.623602 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:06.623612 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.625340 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.625343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:06.625432 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.625455 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 17:25:06.625502 (+ 47us) client_negotiation.cc:240] Negotiation successful
0923 17:25:06.625561 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260923 17:25:06.626289 19561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.619794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38636 (local address 127.11.229.3:35491)
0923 17:25:06.620161 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:06.620171 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:06.620192 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:25:06.620289 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:06.620295 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:06.620553 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:06.621044 (+ 491us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:06.621176 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.622349 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.623745 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.625182 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.625878 (+ 696us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.626017 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:06.626063 (+ 46us) server_negotiation.cc:299] Negotiation successful
0923 17:25:06.626144 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":120,"threads_started":1}
I20260923 17:25:06.628719 19194 tablet_service.cc:1467] Processing CreateTablet for tablet e77c9bff39694e5d90a4cf258dc3debd (DEFAULT_TABLE table=test-table [id=0050cda7a60648dcb57ed2cabc8170c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:25:06.628858 19467 tablet_service.cc:1467] Processing CreateTablet for tablet e77c9bff39694e5d90a4cf258dc3debd (DEFAULT_TABLE table=test-table [id=0050cda7a60648dcb57ed2cabc8170c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:25:06.629899 19467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e77c9bff39694e5d90a4cf258dc3debd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:06.629899 19194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e77c9bff39694e5d90a4cf258dc3debd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:06.633873 19563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.620120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:36405 (local address 127.0.0.1:54478)
0923 17:25:06.621019 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:06.621032 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:06.621117 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:06.622107 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:06.622110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:06.622127 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:06.622222 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:06.622231 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.623745 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.623748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:06.624860 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:06.624870 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.633423 (+ 8553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.633426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:06.633534 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.633554 (+ 20us) client_negotiation.cc:769] Sending connection context
0923 17:25:06.633652 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 17:25:06.633732 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"mutex_wait_us":48,"thread_start_us":95,"threads_started":1}
I20260923 17:25:06.633932 19565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.620518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54478 (local address 127.11.229.2:36405)
0923 17:25:06.621718 (+ 1200us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:06.621733 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:06.621756 (+ 23us) server_negotiation.cc:407] Connection header received
0923 17:25:06.621839 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:06.621843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:06.621980 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:06.622131 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:06.622340 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.623605 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.624998 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.633062 (+ 8064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.633192 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.633440 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:06.633688 (+ 248us) server_negotiation.cc:299] Negotiation successful
0923 17:25:06.633788 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":101,"threads_started":1}
I20260923 17:25:06.636431 19331 tablet_service.cc:1467] Processing CreateTablet for tablet e77c9bff39694e5d90a4cf258dc3debd (DEFAULT_TABLE table=test-table [id=0050cda7a60648dcb57ed2cabc8170c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:25:06.637446 19331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e77c9bff39694e5d90a4cf258dc3debd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:06.639482 19567 tablet_bootstrap.cc:492] T e77c9bff39694e5d90a4cf258dc3debd P e507acf6b90b45269c0f77c976e485ca: Bootstrap starting.
I20260923 17:25:06.642015 19567 tablet_bootstrap.cc:654] T e77c9bff39694e5d90a4cf258dc3debd P e507acf6b90b45269c0f77c976e485ca: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:06.642988 19567 log.cc:824] T e77c9bff39694e5d90a4cf258dc3debd P e507acf6b90b45269c0f77c976e485ca: Log is configured to *not* fsync() on all Append() calls
I20260923 17:25:06.643812 19566 tablet_bootstrap.cc:492] T e77c9bff39694e5d90a4cf258dc3debd P 59e579024b004c1ebea39d11c0ce6461: Bootstrap starting.
I20260923 17:25:06.645780 19567 tablet_bootstrap.cc:492] T e77c9bff39694e5d90a4cf258dc3debd P e507acf6b90b45269c0f77c976e485ca: No bootstrap required, opened a new log
I20260923 17:25:06.646112 19567 ts_tablet_manager.cc:1397] T e77c9bff39694e5d90a4cf258dc3debd P e507acf6b90b45269c0f77c976e485ca: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260923 17:25:06.646628 19569 tablet_bootstrap.cc:492] T e77c9bff39694e5d90a4cf258dc3debd P 50709fadaa9e44cf830396ddbb1b72a4: Bootstrap starting.
I20260923 17:25:06.646978 19566 tablet_bootstrap.cc:654] T e77c9bff39694e5d90a4cf258dc3debd P 59e579024b004c1ebea39d11c0ce6461: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:06.647869 19566 log.cc:824] T e77c9bff39694e5d90a4cf258dc3debd P 59e579024b004c1ebea39d11c0ce6461: Log is configured to *not* fsync() on all Append() calls
I20260923 17:25:06.648787 19569 tablet_bootstrap.cc:654] T e77c9bff39694e5d90a4cf258dc3debd P 50709fadaa9e44cf830396ddbb1b72a4: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:06.649652 19569 log.cc:824] T e77c9bff39694e5d90a4cf258dc3debd P 50709fadaa9e44cf830396ddbb1b72a4: Log is configured to *not* fsync() on all Append() calls
I20260923 17:25:06.650302 19566 tablet_bootstrap.cc:492] T e77c9bff39694e5d90a4cf258dc3debd P 59e579024b004c1ebea39d11c0ce6461: No bootstrap required, opened a new log
I20260923 17:25:06.650552 19566 ts_tablet_manager.cc:1397] T e77c9bff39694e5d90a4cf258dc3debd P 59e579024b004c1ebea39d11c0ce6461: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260923 17:25:06.650424 19567 raft_consensus.cc:348] T e77c9bff39694e5d90a4cf258dc3debd P e507acf6b90b45269c0f77c976e485ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e579024b004c1ebea39d11c0ce6461" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35491 } } peers { permanent_uuid: "50709fadaa9e44cf830396ddbb1b72a4" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36405 } } peers { permanent_uuid: "e507acf6b90b45269c0f77c976e485ca" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36025 } }
I20260923 17:25:06.650710 19567 raft_consensus.cc:374] T e77c9bff39694e5d90a4cf258dc3debd P e507acf6b90b45269c0f77c976e485ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:06.650768 19567 raft_consensus.cc:729] T e77c9bff39694e5d90a4cf258dc3debd P e507acf6b90b45269c0f77c976e485ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e507acf6b90b45269c0f77c976e485ca, State: Initialized, Role: FOLLOWER
I20260923 17:25:06.651257 19567 consensus_queue.cc:260] T e77c9bff39694e5d90a4cf258dc3debd P e507acf6b90b45269c0f77c976e485ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e579024b004c1ebea39d11c0ce6461" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35491 } } peers { permanent_uuid: "50709fadaa9e44cf830396ddbb1b72a4" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36405 } } peers { permanent_uuid: "e507acf6b90b45269c0f77c976e485ca" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36025 } }
I20260923 17:25:06.651744 19569 tablet_bootstrap.cc:492] T e77c9bff39694e5d90a4cf258dc3debd P 50709fadaa9e44cf830396ddbb1b72a4: No bootstrap required, opened a new log
I20260923 17:25:06.651932 19569 ts_tablet_manager.cc:1397] T e77c9bff39694e5d90a4cf258dc3debd P 50709fadaa9e44cf830396ddbb1b72a4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 17:25:06.652307 19567 ts_tablet_manager.cc:1428] T e77c9bff39694e5d90a4cf258dc3debd P e507acf6b90b45269c0f77c976e485ca: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 17:25:06.652614 19261 heartbeater.cc:503] Master 127.11.229.62:46815 was elected leader, sending a full tablet report...
I20260923 17:25:06.656605 19569 raft_consensus.cc:348] T e77c9bff39694e5d90a4cf258dc3debd P 50709fadaa9e44cf830396ddbb1b72a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e579024b004c1ebea39d11c0ce6461" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35491 } } peers { permanent_uuid: "50709fadaa9e44cf830396ddbb1b72a4" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36405 } } peers { permanent_uuid: "e507acf6b90b45269c0f77c976e485ca" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36025 } }
I20260923 17:25:06.656899 19569 raft_consensus.cc:374] T e77c9bff39694e5d90a4cf258dc3debd P 50709fadaa9e44cf830396ddbb1b72a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:06.656951 19569 raft_consensus.cc:729] T e77c9bff39694e5d90a4cf258dc3debd P 50709fadaa9e44cf830396ddbb1b72a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50709fadaa9e44cf830396ddbb1b72a4, State: Initialized, Role: FOLLOWER
I20260923 17:25:06.657438 19569 consensus_queue.cc:260] T e77c9bff39694e5d90a4cf258dc3debd P 50709fadaa9e44cf830396ddbb1b72a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e579024b004c1ebea39d11c0ce6461" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35491 } } peers { permanent_uuid: "50709fadaa9e44cf830396ddbb1b72a4" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36405 } } peers { permanent_uuid: "e507acf6b90b45269c0f77c976e485ca" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36025 } }
I20260923 17:25:06.658476 19569 ts_tablet_manager.cc:1428] T e77c9bff39694e5d90a4cf258dc3debd P 50709fadaa9e44cf830396ddbb1b72a4: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260923 17:25:06.658994 19398 heartbeater.cc:503] Master 127.11.229.62:46815 was elected leader, sending a full tablet report...
I20260923 17:25:06.659376 19566 raft_consensus.cc:348] T e77c9bff39694e5d90a4cf258dc3debd P 59e579024b004c1ebea39d11c0ce6461 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e579024b004c1ebea39d11c0ce6461" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35491 } } peers { permanent_uuid: "50709fadaa9e44cf830396ddbb1b72a4" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36405 } } peers { permanent_uuid: "e507acf6b90b45269c0f77c976e485ca" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36025 } }
I20260923 17:25:06.659646 19566 raft_consensus.cc:374] T e77c9bff39694e5d90a4cf258dc3debd P 59e579024b004c1ebea39d11c0ce6461 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:06.659711 19566 raft_consensus.cc:729] T e77c9bff39694e5d90a4cf258dc3debd P 59e579024b004c1ebea39d11c0ce6461 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59e579024b004c1ebea39d11c0ce6461, State: Initialized, Role: FOLLOWER
I20260923 17:25:06.660687 19566 consensus_queue.cc:260] T e77c9bff39694e5d90a4cf258dc3debd P 59e579024b004c1ebea39d11c0ce6461 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e579024b004c1ebea39d11c0ce6461" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35491 } } peers { permanent_uuid: "50709fadaa9e44cf830396ddbb1b72a4" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36405 } } peers { permanent_uuid: "e507acf6b90b45269c0f77c976e485ca" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36025 } }
I20260923 17:25:06.661903 19566 ts_tablet_manager.cc:1428] T e77c9bff39694e5d90a4cf258dc3debd P 59e579024b004c1ebea39d11c0ce6461: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260923 17:25:06.662258 19534 heartbeater.cc:503] Master 127.11.229.62:46815 was elected leader, sending a full tablet report...
W20260923 17:25:06.674085 19263 tablet.cc:2369] T e77c9bff39694e5d90a4cf258dc3debd P e507acf6b90b45269c0f77c976e485ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:25:06.753576 19572 raft_consensus.cc:482] T e77c9bff39694e5d90a4cf258dc3debd P e507acf6b90b45269c0f77c976e485ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:25:06.753825 19572 raft_consensus.cc:504] T e77c9bff39694e5d90a4cf258dc3debd P e507acf6b90b45269c0f77c976e485ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e579024b004c1ebea39d11c0ce6461" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35491 } } peers { permanent_uuid: "50709fadaa9e44cf830396ddbb1b72a4" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36405 } } peers { permanent_uuid: "e507acf6b90b45269c0f77c976e485ca" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36025 } }
I20260923 17:25:06.755101 19572 leader_election.cc:288] T e77c9bff39694e5d90a4cf258dc3debd P e507acf6b90b45269c0f77c976e485ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59e579024b004c1ebea39d11c0ce6461 (127.11.229.3:35491), 50709fadaa9e44cf830396ddbb1b72a4 (127.11.229.2:36405)
I20260923 17:25:06.759121 19565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.755861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:35923 (local address 127.11.229.2:36405)
0923 17:25:06.756001 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:06.756004 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:06.756016 (+ 12us) server_negotiation.cc:407] Connection header received
0923 17:25:06.756090 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:06.756093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:06.756138 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:06.756215 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:06.756542 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.757052 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.758137 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.758540 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.758657 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.758716 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:06.758921 (+ 205us) server_negotiation.cc:299] Negotiation successful
0923 17:25:06.758990 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 17:25:06.759135 19561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.755461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:46229 (local address 127.11.229.3:35491)
0923 17:25:06.755598 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:06.755603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:06.755673 (+ 70us) server_negotiation.cc:407] Connection header received
0923 17:25:06.755913 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:06.755920 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:06.755975 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:06.756046 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:06.756374 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.756941 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.758182 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.758556 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.758698 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.758752 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:06.758957 (+ 205us) server_negotiation.cc:299] Negotiation successful
0923 17:25:06.759025 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:25:06.759140 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.755302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:35491 (local address 127.11.229.1:46229)
0923 17:25:06.755579 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:06.755599 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:06.755778 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:06.756112 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:06.756115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:06.756134 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:06.756253 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:06.756260 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.757084 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.757090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:06.758025 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:06.758035 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.758708 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.758712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:06.758810 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.758828 (+ 18us) client_negotiation.cc:769] Sending connection context
0923 17:25:06.758921 (+ 93us) client_negotiation.cc:240] Negotiation successful
0923 17:25:06.758990 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":25,"spinlock_wait_cycles":5888,"thread_start_us":118,"threads_started":1}
I20260923 17:25:06.759150 19576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.755443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:36405 (local address 127.11.229.1:35923)
0923 17:25:06.755675 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:06.755688 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:06.755778 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:06.756257 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:06.756260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:06.756281 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:06.756418 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:06.756425 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.757164 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.757167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:06.758007 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:06.758016 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.758672 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.758676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:06.758779 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.758802 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 17:25:06.758873 (+ 71us) client_negotiation.cc:240] Negotiation successful
0923 17:25:06.758973 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":4,"spinlock_wait_cycles":5760,"thread_start_us":75,"threads_started":1}
I20260923 17:25:06.759864 19351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e77c9bff39694e5d90a4cf258dc3debd" candidate_uuid: "e507acf6b90b45269c0f77c976e485ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50709fadaa9e44cf830396ddbb1b72a4" is_pre_election: true
I20260923 17:25:06.759959 19487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e77c9bff39694e5d90a4cf258dc3debd" candidate_uuid: "e507acf6b90b45269c0f77c976e485ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59e579024b004c1ebea39d11c0ce6461" is_pre_election: true
I20260923 17:25:06.760210 19487 raft_consensus.cc:2454] T e77c9bff39694e5d90a4cf258dc3debd P 59e579024b004c1ebea39d11c0ce6461 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e507acf6b90b45269c0f77c976e485ca in term 0.
I20260923 17:25:06.760210 19351 raft_consensus.cc:2454] T e77c9bff39694e5d90a4cf258dc3debd P 50709fadaa9e44cf830396ddbb1b72a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e507acf6b90b45269c0f77c976e485ca in term 0.
I20260923 17:25:06.760701 19150 leader_election.cc:302] T e77c9bff39694e5d90a4cf258dc3debd P e507acf6b90b45269c0f77c976e485ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59e579024b004c1ebea39d11c0ce6461, e507acf6b90b45269c0f77c976e485ca; no voters:
I20260923 17:25:06.760988 19572 raft_consensus.cc:2781] T e77c9bff39694e5d90a4cf258dc3debd P e507acf6b90b45269c0f77c976e485ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:25:06.761098 19572 raft_consensus.cc:482] T e77c9bff39694e5d90a4cf258dc3debd P e507acf6b90b45269c0f77c976e485ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:25:06.761130 19572 raft_consensus.cc:3037] T e77c9bff39694e5d90a4cf258dc3debd P e507acf6b90b45269c0f77c976e485ca [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:06.762223 19572 raft_consensus.cc:504] T e77c9bff39694e5d90a4cf258dc3debd P e507acf6b90b45269c0f77c976e485ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e579024b004c1ebea39d11c0ce6461" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35491 } } peers { permanent_uuid: "50709fadaa9e44cf830396ddbb1b72a4" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36405 } } peers { permanent_uuid: "e507acf6b90b45269c0f77c976e485ca" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36025 } }
I20260923 17:25:06.762581 19572 leader_election.cc:288] T e77c9bff39694e5d90a4cf258dc3debd P e507acf6b90b45269c0f77c976e485ca [CANDIDATE]: Term 1 election: Requested vote from peers 59e579024b004c1ebea39d11c0ce6461 (127.11.229.3:35491), 50709fadaa9e44cf830396ddbb1b72a4 (127.11.229.2:36405)
I20260923 17:25:06.763082 19351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e77c9bff39694e5d90a4cf258dc3debd" candidate_uuid: "e507acf6b90b45269c0f77c976e485ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50709fadaa9e44cf830396ddbb1b72a4"
I20260923 17:25:06.763082 19487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e77c9bff39694e5d90a4cf258dc3debd" candidate_uuid: "e507acf6b90b45269c0f77c976e485ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59e579024b004c1ebea39d11c0ce6461"
I20260923 17:25:06.763216 19351 raft_consensus.cc:3037] T e77c9bff39694e5d90a4cf258dc3debd P 50709fadaa9e44cf830396ddbb1b72a4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:06.763216 19487 raft_consensus.cc:3037] T e77c9bff39694e5d90a4cf258dc3debd P 59e579024b004c1ebea39d11c0ce6461 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:06.764544 19487 raft_consensus.cc:2454] T e77c9bff39694e5d90a4cf258dc3debd P 59e579024b004c1ebea39d11c0ce6461 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e507acf6b90b45269c0f77c976e485ca in term 1.
I20260923 17:25:06.764544 19351 raft_consensus.cc:2454] T e77c9bff39694e5d90a4cf258dc3debd P 50709fadaa9e44cf830396ddbb1b72a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e507acf6b90b45269c0f77c976e485ca in term 1.
I20260923 17:25:06.764848 19148 leader_election.cc:302] T e77c9bff39694e5d90a4cf258dc3debd P e507acf6b90b45269c0f77c976e485ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50709fadaa9e44cf830396ddbb1b72a4, e507acf6b90b45269c0f77c976e485ca; no voters:
I20260923 17:25:06.765064 19572 raft_consensus.cc:2781] T e77c9bff39694e5d90a4cf258dc3debd P e507acf6b90b45269c0f77c976e485ca [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:25:06.765319 19572 raft_consensus.cc:686] T e77c9bff39694e5d90a4cf258dc3debd P e507acf6b90b45269c0f77c976e485ca [term 1 LEADER]: Becoming Leader. State: Replica: e507acf6b90b45269c0f77c976e485ca, State: Running, Role: LEADER
I20260923 17:25:06.765697 19572 consensus_queue.cc:237] T e77c9bff39694e5d90a4cf258dc3debd P e507acf6b90b45269c0f77c976e485ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e579024b004c1ebea39d11c0ce6461" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35491 } } peers { permanent_uuid: "50709fadaa9e44cf830396ddbb1b72a4" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36405 } } peers { permanent_uuid: "e507acf6b90b45269c0f77c976e485ca" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36025 } }
I20260923 17:25:06.768904 18296 catalog_manager.cc:5495] T e77c9bff39694e5d90a4cf258dc3debd P e507acf6b90b45269c0f77c976e485ca reported cstate change: term changed from 0 to 1, leader changed from <none> to e507acf6b90b45269c0f77c976e485ca (127.11.229.1). New cstate: current_term: 1 leader_uuid: "e507acf6b90b45269c0f77c976e485ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e579024b004c1ebea39d11c0ce6461" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50709fadaa9e44cf830396ddbb1b72a4" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e507acf6b90b45269c0f77c976e485ca" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36025 } health_report { overall_health: HEALTHY } } }
I20260923 17:25:06.814663 19564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.811120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41222 (local address 127.11.229.1:36025)
0923 17:25:06.811252 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:06.811256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:06.811270 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:25:06.811421 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:06.811424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:06.811481 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:06.811573 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:25:06.811787 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.812407 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.813244 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.813503 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.813581 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.813844 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:25:06.814274 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:25:06.814364 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:06.814500 (+ 136us) server_negotiation.cc:299] Negotiation successful
0923 17:25:06.814552 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 17:25:06.821033 19351 raft_consensus.cc:1261] T e77c9bff39694e5d90a4cf258dc3debd P 50709fadaa9e44cf830396ddbb1b72a4 [term 1 FOLLOWER]: Refusing update from remote peer e507acf6b90b45269c0f77c976e485ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:25:06.821033 19487 raft_consensus.cc:1261] T e77c9bff39694e5d90a4cf258dc3debd P 59e579024b004c1ebea39d11c0ce6461 [term 1 FOLLOWER]: Refusing update from remote peer e507acf6b90b45269c0f77c976e485ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:25:06.821918 19572 consensus_queue.cc:1035] T e77c9bff39694e5d90a4cf258dc3debd P e507acf6b90b45269c0f77c976e485ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "50709fadaa9e44cf830396ddbb1b72a4" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:25:06.822139 19577 consensus_queue.cc:1035] T e77c9bff39694e5d90a4cf258dc3debd P e507acf6b90b45269c0f77c976e485ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "59e579024b004c1ebea39d11c0ce6461" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 17:25:06.828136 19536 tablet.cc:2369] T e77c9bff39694e5d90a4cf258dc3debd P 59e579024b004c1ebea39d11c0ce6461: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:25:06.831849 19582 mvcc.cc:205] Tried to move back new op lower bound from 7332594920730607616 to 7332594920515096576. Current Snapshot: MvccSnapshot[applied={T|T < 7332594920730607616}]
I20260923 17:25:06.833103 19581 mvcc.cc:205] Tried to move back new op lower bound from 7332594920730607616 to 7332594920515096576. Current Snapshot: MvccSnapshot[applied={T|T < 7332594920730607616}]
I20260923 17:25:06.845012 18297 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:25:06.845206 18297 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:25:06.849867 18297 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5536110589147d9b37d53ea18ab3f26: Sending DeleteTablet for 3 replicas of tablet e77c9bff39694e5d90a4cf258dc3debd
I20260923 17:25:06.850665 19467 tablet_service.cc:1514] Processing DeleteTablet for tablet e77c9bff39694e5d90a4cf258dc3debd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:25:06 UTC) from {username='oryx', principal='oryx/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:38636
I20260923 17:25:06.850872 19331 tablet_service.cc:1514] Processing DeleteTablet for tablet e77c9bff39694e5d90a4cf258dc3debd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:25:06 UTC) from {username='oryx', principal='oryx/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:54478
I20260923 17:25:06.851177 19590 tablet_replica.cc:330] T e77c9bff39694e5d90a4cf258dc3debd P 59e579024b004c1ebea39d11c0ce6461: stopping tablet replica
I20260923 17:25:06.851202 19194 tablet_service.cc:1514] Processing DeleteTablet for tablet e77c9bff39694e5d90a4cf258dc3debd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:25:06 UTC) from {username='oryx', principal='oryx/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:41218
I20260923 17:25:06.851562 19590 raft_consensus.cc:2229] T e77c9bff39694e5d90a4cf258dc3debd P 59e579024b004c1ebea39d11c0ce6461 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:25:06.851899 19590 raft_consensus.cc:2258] T e77c9bff39694e5d90a4cf258dc3debd P 59e579024b004c1ebea39d11c0ce6461 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:25:06.853695 19591 tablet_replica.cc:330] T e77c9bff39694e5d90a4cf258dc3debd P 50709fadaa9e44cf830396ddbb1b72a4: stopping tablet replica
I20260923 17:25:06.853870 19590 ts_tablet_manager.cc:1905] T e77c9bff39694e5d90a4cf258dc3debd P 59e579024b004c1ebea39d11c0ce6461: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:25:06.854115 19591 raft_consensus.cc:2229] T e77c9bff39694e5d90a4cf258dc3debd P 50709fadaa9e44cf830396ddbb1b72a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:25:06.854249 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 18265
I20260923 17:25:06.855265 19591 raft_consensus.cc:2258] T e77c9bff39694e5d90a4cf258dc3debd P 50709fadaa9e44cf830396ddbb1b72a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:25:06.856882 19591 ts_tablet_manager.cc:1905] T e77c9bff39694e5d90a4cf258dc3debd P 50709fadaa9e44cf830396ddbb1b72a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:25:06.857535 19590 ts_tablet_manager.cc:1918] T e77c9bff39694e5d90a4cf258dc3debd P 59e579024b004c1ebea39d11c0ce6461: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:25:06.857641 19590 log.cc:1196] T e77c9bff39694e5d90a4cf258dc3debd P 59e579024b004c1ebea39d11c0ce6461: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-2/wal/wals/e77c9bff39694e5d90a4cf258dc3debd
I20260923 17:25:06.857977 19590 ts_tablet_manager.cc:1939] T e77c9bff39694e5d90a4cf258dc3debd P 59e579024b004c1ebea39d11c0ce6461: Deleting consensus metadata
I20260923 17:25:06.859346 19592 tablet_replica.cc:330] T e77c9bff39694e5d90a4cf258dc3debd P e507acf6b90b45269c0f77c976e485ca: stopping tablet replica
I20260923 17:25:06.859680 19592 raft_consensus.cc:2229] T e77c9bff39694e5d90a4cf258dc3debd P e507acf6b90b45269c0f77c976e485ca [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:25:06.859838 19591 ts_tablet_manager.cc:1918] T e77c9bff39694e5d90a4cf258dc3debd P 50709fadaa9e44cf830396ddbb1b72a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:25:06.859966 19591 log.cc:1196] T e77c9bff39694e5d90a4cf258dc3debd P 50709fadaa9e44cf830396ddbb1b72a4: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-1/wal/wals/e77c9bff39694e5d90a4cf258dc3debd
I20260923 17:25:06.860106 19592 raft_consensus.cc:2258] T e77c9bff39694e5d90a4cf258dc3debd P e507acf6b90b45269c0f77c976e485ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:25:06.860370 19591 ts_tablet_manager.cc:1939] T e77c9bff39694e5d90a4cf258dc3debd P 50709fadaa9e44cf830396ddbb1b72a4: Deleting consensus metadata
I20260923 17:25:06.861682 19592 ts_tablet_manager.cc:1905] T e77c9bff39694e5d90a4cf258dc3debd P e507acf6b90b45269c0f77c976e485ca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260923 17:25:06.863080 19285 connection.cc:535] server connection from 127.0.0.1:54478 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260923 17:25:06.863348 19423 connection.cc:535] server connection from 127.0.0.1:38636 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 17:25:06.864821 19592 ts_tablet_manager.cc:1918] T e77c9bff39694e5d90a4cf258dc3debd P e507acf6b90b45269c0f77c976e485ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:25:06.864923 19592 log.cc:1196] T e77c9bff39694e5d90a4cf258dc3debd P e507acf6b90b45269c0f77c976e485ca: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-0/wal/wals/e77c9bff39694e5d90a4cf258dc3debd
I20260923 17:25:06.865281 19592 ts_tablet_manager.cc:1939] T e77c9bff39694e5d90a4cf258dc3debd P e507acf6b90b45269c0f77c976e485ca: Deleting consensus metadata
W20260923 17:25:06.866261 19150 connection.cc:441] server connection from 127.0.0.1:41218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41218 (request call id 1) could send its response
I20260923 17:25:06.875233 19546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.853488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52206 (local address 127.11.229.61:41129)
0923 17:25:06.853597 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:06.853602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:06.853616 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:25:06.853653 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:06.853656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:06.853699 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:06.853768 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:06.865692 (+ 11924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.866467 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.867681 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.868050 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.868165 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.870511 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:06.870515 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:06.870518 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:06.870546 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:06.872813 (+ 2267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:06.873403 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:06.873408 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:06.873412 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:06.873499 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:06.873797 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:06.873800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:06.873801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:06.874045 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:06.874180 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:06.874871 (+ 691us) server_negotiation.cc:299] Negotiation successful
0923 17:25:06.875025 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260923 17:25:06.877612 19267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.866875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42818 (local address 127.11.229.60:45961)
0923 17:25:06.867014 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:06.867019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:06.867270 (+ 251us) server_negotiation.cc:407] Connection header received
0923 17:25:06.867436 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:06.867439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:06.867501 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:06.867553 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:06.868436 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.869194 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.870493 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.870894 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.870977 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.873750 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:06.873756 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:06.873762 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:06.873791 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:06.875705 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:06.876188 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:06.876193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:06.876195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:06.876280 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:06.876601 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:06.876606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:06.876610 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:06.876818 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:06.876931 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:06.877332 (+ 401us) server_negotiation.cc:299] Negotiation successful
0923 17:25:06.877451 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 17:25:06.879748 19020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.869632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52210 (local address 127.11.229.61:41129)
0923 17:25:06.869741 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:06.869744 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:06.870512 (+ 768us) server_negotiation.cc:407] Connection header received
0923 17:25:06.870700 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:06.870703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:06.870767 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:06.870815 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:06.871687 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.872510 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.873694 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.874026 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.874095 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.875973 (+ 1878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:06.875977 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:06.875980 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:06.876004 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:06.877665 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:06.878355 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:06.878358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:06.878360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:06.878437 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:06.878778 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:06.878782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:06.878783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:06.878973 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:06.879067 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:06.879414 (+ 347us) server_negotiation.cc:299] Negotiation successful
0923 17:25:06.879519 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 17:25:06.885115 19548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:06.853150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42816 (local address 127.11.229.60:45961)
0923 17:25:06.853369 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:06.853374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:06.876343 (+ 22969us) server_negotiation.cc:407] Connection header received
0923 17:25:06.876504 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:06.876508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:06.876566 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:06.876618 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:06.877432 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.878122 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.879188 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:06.879537 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:06.879650 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:06.881163 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:06.881166 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:06.881169 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:06.881192 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:06.883439 (+ 2247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:06.883897 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:06.883902 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:06.883904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:06.883991 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:06.884271 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:06.884276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:06.884278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:06.884449 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:06.884583 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:06.884817 (+ 234us) server_negotiation.cc:299] Negotiation successful
0923 17:25:06.884992 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260923 17:25:07.835690 19601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:07.835065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46815 (local address 127.11.229.2:59467)
0923 17:25:07.835460 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:07.835565 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":82,"threads_started":1}
W20260923 17:25:07.836077 19398 heartbeater.cc:644] Failed to heartbeat to 127.11.229.62:46815 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.229.62:46815: connect: Connection refused (error 111)
I20260923 17:25:07.836155 19602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:07.835522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46815 (local address 127.11.229.3:43355)
0923 17:25:07.835778 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:07.836053 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":64,"threads_started":1}
W20260923 17:25:07.836447 19534 heartbeater.cc:644] Failed to heartbeat to 127.11.229.62:46815 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.229.62:46815: connect: Connection refused (error 111)
I20260923 17:25:07.837635 19603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:07.837264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46815 (local address 127.11.229.1:33779)
0923 17:25:07.837488 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:07.837559 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":68,"threads_started":1}
W20260923 17:25:07.837920 19261 heartbeater.cc:644] Failed to heartbeat to 127.11.229.62:46815 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.229.62:46815: connect: Connection refused (error 111)
I20260923 17:25:08.350433 19604 raft_consensus.cc:482] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b5536110589147d9b37d53ea18ab3f26)
I20260923 17:25:08.350625 19604 raft_consensus.cc:504] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: false } } peers { permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45961 } attrs { promote: false } }
I20260923 17:25:08.351598 19604 leader_election.cc:288] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b5536110589147d9b37d53ea18ab3f26 (127.11.229.62:46815), 23b480ef1799433abb57f326cf3078be (127.11.229.61:41129)
I20260923 17:25:08.352046 19606 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b5536110589147d9b37d53ea18ab3f26)
I20260923 17:25:08.352321 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.351800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46815 (local address 127.0.0.1:45422)
0923 17:25:08.352128 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:08.352215 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":62,"threads_started":1}
I20260923 17:25:08.352298 19606 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: false } } peers { permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45961 } attrs { promote: false } }
W20260923 17:25:08.352543 19066 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.229.62:46815: connect: Connection refused (error 111)
I20260923 17:25:08.353454 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.353198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46815 (local address 127.0.0.1:45430)
0923 17:25:08.353338 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:08.353390 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260923 17:25:08.353698 19066 leader_election.cc:334] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b5536110589147d9b37d53ea18ab3f26 (127.11.229.62:46815): Network error: Client connection negotiation failed: client connection to 127.11.229.62:46815: connect: Connection refused (error 111)
I20260923 17:25:08.353744 19606 leader_election.cc:288] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b5536110589147d9b37d53ea18ab3f26 (127.11.229.62:46815), b163f7eb24d54419aa3e1337a7c013ae (127.11.229.60:45961)
I20260923 17:25:08.354178 19610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.353439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46815 (local address 127.0.0.1:45432)
0923 17:25:08.354022 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:08.354092 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":83,"threads_started":1}
W20260923 17:25:08.354410 18883 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.229.62:46815: connect: Connection refused (error 111)
I20260923 17:25:08.355278 19611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.355023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46815 (local address 127.0.0.1:45436)
0923 17:25:08.355129 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:08.355178 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260923 17:25:08.355412 18883 leader_election.cc:334] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b5536110589147d9b37d53ea18ab3f26 (127.11.229.62:46815): Network error: Client connection negotiation failed: client connection to 127.11.229.62:46815: connect: Connection refused (error 111)
I20260923 17:25:08.356266 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.351831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:41129 (local address 127.0.0.1:52224)
0923 17:25:08.352148 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:08.352167 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:08.352283 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:08.352591 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:08.352594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:08.352611 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:08.352728 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:08.352734 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:08.353847 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:08.353853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:08.355141 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:08.355148 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:08.355965 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:08.355968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:08.356043 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:08.356062 (+ 19us) client_negotiation.cc:769] Sending connection context
0923 17:25:08.356121 (+ 59us) client_negotiation.cc:240] Negotiation successful
0923 17:25:08.356175 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":74,"threads_started":1}
I20260923 17:25:08.356351 19608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.352017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52224 (local address 127.11.229.61:41129)
0923 17:25:08.352261 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:08.352266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:08.352281 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:25:08.352406 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:08.352408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:08.352455 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:08.352570 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:08.352891 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:08.353693 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:08.355256 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:08.355854 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:08.355957 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:08.356017 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:08.356163 (+ 146us) server_negotiation.cc:299] Negotiation successful
0923 17:25:08.356239 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":64,"threads_started":1}
I20260923 17:25:08.356756 18908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b163f7eb24d54419aa3e1337a7c013ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "23b480ef1799433abb57f326cf3078be" is_pre_election: true
I20260923 17:25:08.356968 18908 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b163f7eb24d54419aa3e1337a7c013ae in term 1.
I20260923 17:25:08.357365 19065 leader_election.cc:302] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23b480ef1799433abb57f326cf3078be, b163f7eb24d54419aa3e1337a7c013ae; no voters: b5536110589147d9b37d53ea18ab3f26
I20260923 17:25:08.357594 19604 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260923 17:25:08.357681 19604 raft_consensus.cc:482] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [term 1 FOLLOWER]: Starting leader election (detected failure of leader b5536110589147d9b37d53ea18ab3f26)
I20260923 17:25:08.357734 19604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [term 1 FOLLOWER]: Advancing to term 2
I20260923 17:25:08.358312 19612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.354161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42820 (local address 127.11.229.60:45961)
0923 17:25:08.354382 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:08.354390 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:08.354403 (+ 13us) server_negotiation.cc:407] Connection header received
0923 17:25:08.354581 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:08.354584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:08.354626 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:08.354944 (+ 318us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:08.355021 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:08.355718 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:08.357204 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:08.357799 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:08.357881 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:08.357923 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:08.358169 (+ 246us) server_negotiation.cc:299] Negotiation successful
0923 17:25:08.358228 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":164,"thread_start_us":87,"threads_started":1}
I20260923 17:25:08.358322 19610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.353930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:45961 (local address 127.0.0.1:42820)
0923 17:25:08.354338 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:08.354352 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:08.354443 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:08.354744 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:08.354747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:08.354764 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:08.354853 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:08.354859 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:08.355828 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:08.355831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:08.357076 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:08.357089 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:08.357915 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:08.357922 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:08.358023 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:08.358058 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 17:25:08.358141 (+ 83us) client_negotiation.cc:240] Negotiation successful
0923 17:25:08.358212 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":73,"threads_started":1}
I20260923 17:25:08.358717 19094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "23b480ef1799433abb57f326cf3078be" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b163f7eb24d54419aa3e1337a7c013ae" is_pre_election: true
I20260923 17:25:08.359141 19604 raft_consensus.cc:504] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: false } } peers { permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45961 } attrs { promote: false } }
I20260923 17:25:08.359380 19094 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 23b480ef1799433abb57f326cf3078be in current term 2: Already voted for candidate b163f7eb24d54419aa3e1337a7c013ae in this term.
I20260923 17:25:08.359560 19604 leader_election.cc:288] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [CANDIDATE]: Term 2 election: Requested vote from peers b5536110589147d9b37d53ea18ab3f26 (127.11.229.62:46815), 23b480ef1799433abb57f326cf3078be (127.11.229.61:41129)
I20260923 17:25:08.359887 18885 leader_election.cc:302] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23b480ef1799433abb57f326cf3078be; no voters: b163f7eb24d54419aa3e1337a7c013ae, b5536110589147d9b37d53ea18ab3f26
I20260923 17:25:08.360109 19606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [term 1 FOLLOWER]: Advancing to term 2
I20260923 17:25:08.360109 18908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b163f7eb24d54419aa3e1337a7c013ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "23b480ef1799433abb57f326cf3078be"
I20260923 17:25:08.360311 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.359740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46815 (local address 127.0.0.1:45446)
0923 17:25:08.359986 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:08.360242 (+ 256us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
I20260923 17:25:08.360930 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.360704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46815 (local address 127.0.0.1:45450)
0923 17:25:08.360821 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:08.360874 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 17:25:08.361049 19066 leader_election.cc:334] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b5536110589147d9b37d53ea18ab3f26 (127.11.229.62:46815): Network error: Client connection negotiation failed: client connection to 127.11.229.62:46815: connect: Connection refused (error 111)
I20260923 17:25:08.361474 19606 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 17:25:08.362382 18908 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b163f7eb24d54419aa3e1337a7c013ae in term 2.
I20260923 17:25:08.362861 19065 leader_election.cc:302] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23b480ef1799433abb57f326cf3078be, b163f7eb24d54419aa3e1337a7c013ae; no voters: b5536110589147d9b37d53ea18ab3f26
I20260923 17:25:08.363066 19604 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [term 2 FOLLOWER]: Leader election won for term 2
I20260923 17:25:08.363286 19604 raft_consensus.cc:686] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [term 2 LEADER]: Becoming Leader. State: Replica: b163f7eb24d54419aa3e1337a7c013ae, State: Running, Role: LEADER
I20260923 17:25:08.363556 19604 consensus_queue.cc:237] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: false } } peers { permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45961 } attrs { promote: false } }
I20260923 17:25:08.364943 19614 sys_catalog.cc:455] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader b163f7eb24d54419aa3e1337a7c013ae. Latest consensus state: current_term: 2 leader_uuid: "b163f7eb24d54419aa3e1337a7c013ae" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: false } } peers { permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45961 } attrs { promote: false } } }
I20260923 17:25:08.365088 19614 sys_catalog.cc:458] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [sys.catalog]: This master's current role is: LEADER
I20260923 17:25:08.365438 19616 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:25:08.367515 19616 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:25:08.367858 19616 catalog_manager.cc:1259] Loaded cluster ID: 78a04d62a08a462dab6b3c54b4731f38
I20260923 17:25:08.367901 19616 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:25:08.368391 19616 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:25:08.368734 19616 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae: Loaded TSK: 0
I20260923 17:25:08.369488 19616 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:25:08.395741 19084 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:25:08.399609 19084 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:25:08.402774 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.402509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46815 (local address 127.0.0.1:45466)
0923 17:25:08.402646 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:08.402707 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260923 17:25:08.402851 18908 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [term 2 FOLLOWER]: Refusing update from remote peer b163f7eb24d54419aa3e1337a7c013ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260923 17:25:08.403527 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.403286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46815 (local address 127.0.0.1:45470)
0923 17:25:08.403391 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:08.403444 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260923 17:25:08.403445 19614 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260923 17:25:08.403713 19066 consensus_peers.cc:487] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae -> Peer b5536110589147d9b37d53ea18ab3f26 (127.11.229.62:46815): Couldn't send request to peer b5536110589147d9b37d53ea18ab3f26. Status: Network error: Client connection negotiation failed: client connection to 127.11.229.62:46815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260923 17:25:08.405097 19606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [sys.catalog]: SysCatalogTable state changed. Reason: New leader b163f7eb24d54419aa3e1337a7c013ae. Latest consensus state: current_term: 2 leader_uuid: "b163f7eb24d54419aa3e1337a7c013ae" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: false } } peers { permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45961 } attrs { promote: false } } }
I20260923 17:25:08.405238 19606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:25:08.406363 19604 sys_catalog.cc:455] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "b163f7eb24d54419aa3e1337a7c013ae" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: false } } peers { permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45961 } attrs { promote: false } } }
I20260923 17:25:08.406492 19604 sys_catalog.cc:458] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae [sys.catalog]: This master's current role is: LEADER
I20260923 17:25:08.406688 19606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "b163f7eb24d54419aa3e1337a7c013ae" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b5536110589147d9b37d53ea18ab3f26" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 46815 } } peers { permanent_uuid: "23b480ef1799433abb57f326cf3078be" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41129 } attrs { promote: false } } peers { permanent_uuid: "b163f7eb24d54419aa3e1337a7c013ae" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45961 } attrs { promote: false } } }
I20260923 17:25:08.406822 19606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b480ef1799433abb57f326cf3078be [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:25:08.417416 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.412748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:36025 (local address 127.0.0.1:41228)
0923 17:25:08.412894 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:08.412907 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:08.413006 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:08.413623 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:08.413626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:08.413641 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:08.413774 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:08.413781 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:08.414938 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:08.414941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:08.416136 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:08.416146 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:08.417030 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:08.417033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:08.417106 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:08.417147 (+ 41us) client_negotiation.cc:769] Sending connection context
0923 17:25:08.417212 (+ 65us) client_negotiation.cc:240] Negotiation successful
0923 17:25:08.417287 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":3}
I20260923 17:25:08.417416 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.412963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:35491 (local address 127.0.0.1:38652)
0923 17:25:08.413101 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:08.413117 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:08.413203 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:08.413777 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:08.413780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:08.413796 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:08.413901 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:08.413907 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:08.414987 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:08.414992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:08.416006 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:08.416036 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:08.416956 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:08.416963 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:08.417072 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:08.417099 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 17:25:08.417149 (+ 50us) client_negotiation.cc:240] Negotiation successful
0923 17:25:08.417241 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":19}
I20260923 17:25:08.417479 19623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.412847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41228 (local address 127.11.229.1:36025)
0923 17:25:08.413358 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:08.413364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:08.413381 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:25:08.413439 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:08.413442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:08.413494 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:08.413882 (+ 388us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:08.414005 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:08.414823 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:08.416293 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:08.416934 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:08.417027 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:08.417093 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:08.417239 (+ 146us) server_negotiation.cc:299] Negotiation successful
0923 17:25:08.417345 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":106,"threads_started":1}
I20260923 17:25:08.417956 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.413256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38652 (local address 127.11.229.3:35491)
0923 17:25:08.413529 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:08.413535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:08.413551 (+ 16us) server_negotiation.cc:407] Connection header received
0923 17:25:08.413607 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:08.413610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:08.413665 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:08.414001 (+ 336us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:08.414102 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:08.414849 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:08.416183 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:08.416825 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:08.417617 (+ 792us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:08.417696 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:08.417737 (+ 41us) server_negotiation.cc:299] Negotiation successful
0923 17:25:08.417814 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":67,"threads_started":1}
I20260923 17:25:08.418140 19194 tablet_service.cc:1467] Processing CreateTablet for tablet a70d2c2f94ca4bb5bfbc96a00064d20d (DEFAULT_TABLE table=test-table [id=03bd9c04a68f49eab5305af6dff78ee7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:25:08.418450 19194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a70d2c2f94ca4bb5bfbc96a00064d20d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:08.418417 19467 tablet_service.cc:1467] Processing CreateTablet for tablet a70d2c2f94ca4bb5bfbc96a00064d20d (DEFAULT_TABLE table=test-table [id=03bd9c04a68f49eab5305af6dff78ee7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:25:08.418646 19467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a70d2c2f94ca4bb5bfbc96a00064d20d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:08.418644 19626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.413616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54494 (local address 127.11.229.2:36405)
0923 17:25:08.414334 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:08.414337 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:08.414348 (+ 11us) server_negotiation.cc:407] Connection header received
0923 17:25:08.414393 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:08.414395 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:08.414431 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:08.414525 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:08.414765 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:08.415531 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:08.416950 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:08.418074 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:08.418227 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:08.418288 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:08.418459 (+ 171us) server_negotiation.cc:299] Negotiation successful
0923 17:25:08.418527 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":72,"threads_started":1}
I20260923 17:25:08.418648 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.413282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:36405 (local address 127.0.0.1:54494)
0923 17:25:08.414036 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:08.414050 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:08.414140 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:08.414543 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:08.414546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:08.414562 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:08.414652 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:08.414657 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:08.415655 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:08.415658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:08.416802 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:08.416812 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:08.418230 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:08.418234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:08.418322 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:08.418342 (+ 20us) client_negotiation.cc:769] Sending connection context
0923 17:25:08.418442 (+ 100us) client_negotiation.cc:240] Negotiation successful
0923 17:25:08.418510 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":74,"threads_started":1}
I20260923 17:25:08.419198 19331 tablet_service.cc:1467] Processing CreateTablet for tablet a70d2c2f94ca4bb5bfbc96a00064d20d (DEFAULT_TABLE table=test-table [id=03bd9c04a68f49eab5305af6dff78ee7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:25:08.419400 19331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a70d2c2f94ca4bb5bfbc96a00064d20d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:08.421087 19627 tablet_bootstrap.cc:492] T a70d2c2f94ca4bb5bfbc96a00064d20d P 59e579024b004c1ebea39d11c0ce6461: Bootstrap starting.
I20260923 17:25:08.422042 19629 tablet_bootstrap.cc:492] T a70d2c2f94ca4bb5bfbc96a00064d20d P 50709fadaa9e44cf830396ddbb1b72a4: Bootstrap starting.
I20260923 17:25:08.422117 19627 tablet_bootstrap.cc:654] T a70d2c2f94ca4bb5bfbc96a00064d20d P 59e579024b004c1ebea39d11c0ce6461: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:08.422746 19628 tablet_bootstrap.cc:492] T a70d2c2f94ca4bb5bfbc96a00064d20d P e507acf6b90b45269c0f77c976e485ca: Bootstrap starting.
I20260923 17:25:08.422870 19629 tablet_bootstrap.cc:654] T a70d2c2f94ca4bb5bfbc96a00064d20d P 50709fadaa9e44cf830396ddbb1b72a4: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:08.423753 19628 tablet_bootstrap.cc:654] T a70d2c2f94ca4bb5bfbc96a00064d20d P e507acf6b90b45269c0f77c976e485ca: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:08.424047 19629 tablet_bootstrap.cc:492] T a70d2c2f94ca4bb5bfbc96a00064d20d P 50709fadaa9e44cf830396ddbb1b72a4: No bootstrap required, opened a new log
I20260923 17:25:08.424149 19629 ts_tablet_manager.cc:1397] T a70d2c2f94ca4bb5bfbc96a00064d20d P 50709fadaa9e44cf830396ddbb1b72a4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 17:25:08.424613 19627 tablet_bootstrap.cc:492] T a70d2c2f94ca4bb5bfbc96a00064d20d P 59e579024b004c1ebea39d11c0ce6461: No bootstrap required, opened a new log
I20260923 17:25:08.424686 19627 ts_tablet_manager.cc:1397] T a70d2c2f94ca4bb5bfbc96a00064d20d P 59e579024b004c1ebea39d11c0ce6461: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260923 17:25:08.424633 19629 raft_consensus.cc:348] T a70d2c2f94ca4bb5bfbc96a00064d20d P 50709fadaa9e44cf830396ddbb1b72a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e507acf6b90b45269c0f77c976e485ca" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36025 } } peers { permanent_uuid: "59e579024b004c1ebea39d11c0ce6461" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35491 } } peers { permanent_uuid: "50709fadaa9e44cf830396ddbb1b72a4" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36405 } }
I20260923 17:25:08.424774 19629 raft_consensus.cc:374] T a70d2c2f94ca4bb5bfbc96a00064d20d P 50709fadaa9e44cf830396ddbb1b72a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:08.424818 19629 raft_consensus.cc:729] T a70d2c2f94ca4bb5bfbc96a00064d20d P 50709fadaa9e44cf830396ddbb1b72a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50709fadaa9e44cf830396ddbb1b72a4, State: Initialized, Role: FOLLOWER
I20260923 17:25:08.424947 19629 consensus_queue.cc:260] T a70d2c2f94ca4bb5bfbc96a00064d20d P 50709fadaa9e44cf830396ddbb1b72a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e507acf6b90b45269c0f77c976e485ca" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36025 } } peers { permanent_uuid: "59e579024b004c1ebea39d11c0ce6461" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35491 } } peers { permanent_uuid: "50709fadaa9e44cf830396ddbb1b72a4" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36405 } }
I20260923 17:25:08.425163 19627 raft_consensus.cc:348] T a70d2c2f94ca4bb5bfbc96a00064d20d P 59e579024b004c1ebea39d11c0ce6461 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e507acf6b90b45269c0f77c976e485ca" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36025 } } peers { permanent_uuid: "59e579024b004c1ebea39d11c0ce6461" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35491 } } peers { permanent_uuid: "50709fadaa9e44cf830396ddbb1b72a4" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36405 } }
I20260923 17:25:08.425304 19627 raft_consensus.cc:374] T a70d2c2f94ca4bb5bfbc96a00064d20d P 59e579024b004c1ebea39d11c0ce6461 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:08.425329 19629 ts_tablet_manager.cc:1428] T a70d2c2f94ca4bb5bfbc96a00064d20d P 50709fadaa9e44cf830396ddbb1b72a4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 17:25:08.425348 19627 raft_consensus.cc:729] T a70d2c2f94ca4bb5bfbc96a00064d20d P 59e579024b004c1ebea39d11c0ce6461 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59e579024b004c1ebea39d11c0ce6461, State: Initialized, Role: FOLLOWER
I20260923 17:25:08.425488 19627 consensus_queue.cc:260] T a70d2c2f94ca4bb5bfbc96a00064d20d P 59e579024b004c1ebea39d11c0ce6461 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e507acf6b90b45269c0f77c976e485ca" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36025 } } peers { permanent_uuid: "59e579024b004c1ebea39d11c0ce6461" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35491 } } peers { permanent_uuid: "50709fadaa9e44cf830396ddbb1b72a4" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36405 } }
I20260923 17:25:08.425823 19627 ts_tablet_manager.cc:1428] T a70d2c2f94ca4bb5bfbc96a00064d20d P 59e579024b004c1ebea39d11c0ce6461: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260923 17:25:08.426367 19628 tablet_bootstrap.cc:492] T a70d2c2f94ca4bb5bfbc96a00064d20d P e507acf6b90b45269c0f77c976e485ca: No bootstrap required, opened a new log
I20260923 17:25:08.426451 19628 ts_tablet_manager.cc:1397] T a70d2c2f94ca4bb5bfbc96a00064d20d P e507acf6b90b45269c0f77c976e485ca: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260923 17:25:08.426540 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.425778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46815 (local address 127.11.229.2:33257)
0923 17:25:08.426377 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:08.426437 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":90,"threads_started":1}
I20260923 17:25:08.426883 19628 raft_consensus.cc:348] T a70d2c2f94ca4bb5bfbc96a00064d20d P e507acf6b90b45269c0f77c976e485ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e507acf6b90b45269c0f77c976e485ca" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36025 } } peers { permanent_uuid: "59e579024b004c1ebea39d11c0ce6461" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35491 } } peers { permanent_uuid: "50709fadaa9e44cf830396ddbb1b72a4" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36405 } }
I20260923 17:25:08.426995 19628 raft_consensus.cc:374] T a70d2c2f94ca4bb5bfbc96a00064d20d P e507acf6b90b45269c0f77c976e485ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:08.427040 19628 raft_consensus.cc:729] T a70d2c2f94ca4bb5bfbc96a00064d20d P e507acf6b90b45269c0f77c976e485ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e507acf6b90b45269c0f77c976e485ca, State: Initialized, Role: FOLLOWER
I20260923 17:25:08.427181 19628 consensus_queue.cc:260] T a70d2c2f94ca4bb5bfbc96a00064d20d P e507acf6b90b45269c0f77c976e485ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e507acf6b90b45269c0f77c976e485ca" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36025 } } peers { permanent_uuid: "59e579024b004c1ebea39d11c0ce6461" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35491 } } peers { permanent_uuid: "50709fadaa9e44cf830396ddbb1b72a4" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36405 } }
I20260923 17:25:08.427506 19628 ts_tablet_manager.cc:1428] T a70d2c2f94ca4bb5bfbc96a00064d20d P e507acf6b90b45269c0f77c976e485ca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 17:25:08.428057 19397 heartbeater.cc:503] Master 127.11.229.60:45961 was elected leader, sending a full tablet report...
I20260923 17:25:08.428422 19637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.427841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46815 (local address 127.11.229.1:40913)
0923 17:25:08.428254 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:08.428314 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":111,"threads_started":1}
I20260923 17:25:08.429359 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.428936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46815 (local address 127.11.229.1:35965)
0923 17:25:08.429167 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:08.429230 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":99,"threads_started":1}
I20260923 17:25:08.429473 19260 heartbeater.cc:503] Master 127.11.229.60:45961 was elected leader, sending a full tablet report...
I20260923 17:25:08.430789 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.430420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46815 (local address 127.11.229.3:60343)
0923 17:25:08.430650 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:08.430716 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":99,"threads_started":1}
I20260923 17:25:08.431550 19533 heartbeater.cc:503] Master 127.11.229.60:45961 was elected leader, sending a full tablet report...
I20260923 17:25:08.431859 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.431631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46815 (local address 127.11.229.2:35473)
0923 17:25:08.431734 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:08.431787 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260923 17:25:08.432497 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.431535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46815 (local address 127.11.229.3:52907)
0923 17:25:08.432378 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:08.432421 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":784}
W20260923 17:25:08.624588 19400 tablet.cc:2369] T a70d2c2f94ca4bb5bfbc96a00064d20d P 50709fadaa9e44cf830396ddbb1b72a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:25:08.688865 19634 raft_consensus.cc:482] T a70d2c2f94ca4bb5bfbc96a00064d20d P 59e579024b004c1ebea39d11c0ce6461 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:25:08.689020 19634 raft_consensus.cc:504] T a70d2c2f94ca4bb5bfbc96a00064d20d P 59e579024b004c1ebea39d11c0ce6461 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e507acf6b90b45269c0f77c976e485ca" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36025 } } peers { permanent_uuid: "59e579024b004c1ebea39d11c0ce6461" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35491 } } peers { permanent_uuid: "50709fadaa9e44cf830396ddbb1b72a4" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36405 } }
I20260923 17:25:08.690296 19634 leader_election.cc:288] T a70d2c2f94ca4bb5bfbc96a00064d20d P 59e579024b004c1ebea39d11c0ce6461 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e507acf6b90b45269c0f77c976e485ca (127.11.229.1:36025), 50709fadaa9e44cf830396ddbb1b72a4 (127.11.229.2:36405)
I20260923 17:25:08.695554 19623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.690809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:47379 (local address 127.11.229.1:36025)
0923 17:25:08.690954 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:08.690961 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:08.691142 (+ 181us) server_negotiation.cc:407] Connection header received
0923 17:25:08.691205 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:08.691210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:08.691260 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:08.691330 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:08.691574 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:08.692372 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:08.694265 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:08.694897 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:08.695010 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:08.695065 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:08.695338 (+ 273us) server_negotiation.cc:299] Negotiation successful
0923 17:25:08.695416 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 17:25:08.695554 19626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.690710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:59277 (local address 127.11.229.2:36405)
0923 17:25:08.690844 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:08.690849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:08.690864 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:25:08.691053 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:08.691056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:08.691133 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:08.691237 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:08.691574 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:08.692373 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:08.694265 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:08.694877 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:08.695010 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:08.695065 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:08.695338 (+ 273us) server_negotiation.cc:299] Negotiation successful
0923 17:25:08.695416 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 17:25:08.695564 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.690583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:36405 (local address 127.11.229.3:59277)
0923 17:25:08.690738 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:08.690755 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:08.690883 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:08.691264 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:08.691266 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:08.691280 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:08.691430 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:08.691439 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:08.692496 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:08.692499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:08.694133 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:08.694142 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:08.695016 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:08.695020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:08.695194 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:08.695231 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 17:25:08.695303 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 17:25:08.695410 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":201}
I20260923 17:25:08.695575 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.690583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:36025 (local address 127.11.229.3:47379)
0923 17:25:08.691036 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:08.691050 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:08.691132 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:08.691358 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:08.691360 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:08.691372 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:08.691459 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:08.691468 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:08.692496 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:08.692499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:08.694129 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:08.694142 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:08.695016 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:08.695020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:08.695168 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:08.695220 (+ 52us) client_negotiation.cc:769] Sending connection context
0923 17:25:08.695303 (+ 83us) client_negotiation.cc:240] Negotiation successful
0923 17:25:08.695413 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":391,"thread_start_us":128,"threads_started":1}
I20260923 17:25:08.696121 19351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a70d2c2f94ca4bb5bfbc96a00064d20d" candidate_uuid: "59e579024b004c1ebea39d11c0ce6461" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50709fadaa9e44cf830396ddbb1b72a4" is_pre_election: true
I20260923 17:25:08.696300 19351 raft_consensus.cc:2454] T a70d2c2f94ca4bb5bfbc96a00064d20d P 50709fadaa9e44cf830396ddbb1b72a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59e579024b004c1ebea39d11c0ce6461 in term 0.
I20260923 17:25:08.696491 19214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a70d2c2f94ca4bb5bfbc96a00064d20d" candidate_uuid: "59e579024b004c1ebea39d11c0ce6461" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e507acf6b90b45269c0f77c976e485ca" is_pre_election: true
I20260923 17:25:08.696715 19421 leader_election.cc:302] T a70d2c2f94ca4bb5bfbc96a00064d20d P 59e579024b004c1ebea39d11c0ce6461 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50709fadaa9e44cf830396ddbb1b72a4, 59e579024b004c1ebea39d11c0ce6461; no voters:
I20260923 17:25:08.696754 19214 raft_consensus.cc:2454] T a70d2c2f94ca4bb5bfbc96a00064d20d P e507acf6b90b45269c0f77c976e485ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59e579024b004c1ebea39d11c0ce6461 in term 0.
I20260923 17:25:08.697003 19634 raft_consensus.cc:2781] T a70d2c2f94ca4bb5bfbc96a00064d20d P 59e579024b004c1ebea39d11c0ce6461 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:25:08.697085 19634 raft_consensus.cc:482] T a70d2c2f94ca4bb5bfbc96a00064d20d P 59e579024b004c1ebea39d11c0ce6461 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:25:08.697142 19634 raft_consensus.cc:3037] T a70d2c2f94ca4bb5bfbc96a00064d20d P 59e579024b004c1ebea39d11c0ce6461 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:08.698262 19634 raft_consensus.cc:504] T a70d2c2f94ca4bb5bfbc96a00064d20d P 59e579024b004c1ebea39d11c0ce6461 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e507acf6b90b45269c0f77c976e485ca" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36025 } } peers { permanent_uuid: "59e579024b004c1ebea39d11c0ce6461" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35491 } } peers { permanent_uuid: "50709fadaa9e44cf830396ddbb1b72a4" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36405 } }
I20260923 17:25:08.698622 19634 leader_election.cc:288] T a70d2c2f94ca4bb5bfbc96a00064d20d P 59e579024b004c1ebea39d11c0ce6461 [CANDIDATE]: Term 1 election: Requested vote from peers e507acf6b90b45269c0f77c976e485ca (127.11.229.1:36025), 50709fadaa9e44cf830396ddbb1b72a4 (127.11.229.2:36405)
I20260923 17:25:08.698951 19214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a70d2c2f94ca4bb5bfbc96a00064d20d" candidate_uuid: "59e579024b004c1ebea39d11c0ce6461" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e507acf6b90b45269c0f77c976e485ca"
I20260923 17:25:08.699026 19351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a70d2c2f94ca4bb5bfbc96a00064d20d" candidate_uuid: "59e579024b004c1ebea39d11c0ce6461" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50709fadaa9e44cf830396ddbb1b72a4"
I20260923 17:25:08.699074 19214 raft_consensus.cc:3037] T a70d2c2f94ca4bb5bfbc96a00064d20d P e507acf6b90b45269c0f77c976e485ca [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:08.699172 19351 raft_consensus.cc:3037] T a70d2c2f94ca4bb5bfbc96a00064d20d P 50709fadaa9e44cf830396ddbb1b72a4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:08.700093 19214 raft_consensus.cc:2454] T a70d2c2f94ca4bb5bfbc96a00064d20d P e507acf6b90b45269c0f77c976e485ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59e579024b004c1ebea39d11c0ce6461 in term 1.
I20260923 17:25:08.700234 19351 raft_consensus.cc:2454] T a70d2c2f94ca4bb5bfbc96a00064d20d P 50709fadaa9e44cf830396ddbb1b72a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59e579024b004c1ebea39d11c0ce6461 in term 1.
I20260923 17:25:08.700416 19420 leader_election.cc:302] T a70d2c2f94ca4bb5bfbc96a00064d20d P 59e579024b004c1ebea39d11c0ce6461 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59e579024b004c1ebea39d11c0ce6461, e507acf6b90b45269c0f77c976e485ca; no voters:
I20260923 17:25:08.700603 19634 raft_consensus.cc:2781] T a70d2c2f94ca4bb5bfbc96a00064d20d P 59e579024b004c1ebea39d11c0ce6461 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:25:08.700862 19634 raft_consensus.cc:686] T a70d2c2f94ca4bb5bfbc96a00064d20d P 59e579024b004c1ebea39d11c0ce6461 [term 1 LEADER]: Becoming Leader. State: Replica: 59e579024b004c1ebea39d11c0ce6461, State: Running, Role: LEADER
I20260923 17:25:08.701128 19634 consensus_queue.cc:237] T a70d2c2f94ca4bb5bfbc96a00064d20d P 59e579024b004c1ebea39d11c0ce6461 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e507acf6b90b45269c0f77c976e485ca" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36025 } } peers { permanent_uuid: "59e579024b004c1ebea39d11c0ce6461" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35491 } } peers { permanent_uuid: "50709fadaa9e44cf830396ddbb1b72a4" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36405 } }
I20260923 17:25:08.701797 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.701561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46815 (local address 127.11.229.3:42493)
0923 17:25:08.701671 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:08.701732 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260923 17:25:08.703718 19084 catalog_manager.cc:5495] T a70d2c2f94ca4bb5bfbc96a00064d20d P 59e579024b004c1ebea39d11c0ce6461 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59e579024b004c1ebea39d11c0ce6461 (127.11.229.3). New cstate: current_term: 1 leader_uuid: "59e579024b004c1ebea39d11c0ce6461" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e507acf6b90b45269c0f77c976e485ca" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59e579024b004c1ebea39d11c0ce6461" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50709fadaa9e44cf830396ddbb1b72a4" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36405 } health_report { overall_health: UNKNOWN } } }
I20260923 17:25:08.782998 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.779590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38658 (local address 127.11.229.3:35491)
0923 17:25:08.779768 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:08.779771 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:08.779786 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:25:08.779869 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:08.779871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:08.779929 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:08.780038 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:25:08.780270 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:08.780774 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:08.781579 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:08.781787 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:08.781861 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:08.782095 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:25:08.782556 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:25:08.782642 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:08.782760 (+ 118us) server_negotiation.cc:299] Negotiation successful
0923 17:25:08.782811 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260923 17:25:08.786180 19351 raft_consensus.cc:1261] T a70d2c2f94ca4bb5bfbc96a00064d20d P 50709fadaa9e44cf830396ddbb1b72a4 [term 1 FOLLOWER]: Refusing update from remote peer 59e579024b004c1ebea39d11c0ce6461: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:25:08.786463 19214 raft_consensus.cc:1261] T a70d2c2f94ca4bb5bfbc96a00064d20d P e507acf6b90b45269c0f77c976e485ca [term 1 FOLLOWER]: Refusing update from remote peer 59e579024b004c1ebea39d11c0ce6461: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:25:08.786777 19641 consensus_queue.cc:1035] T a70d2c2f94ca4bb5bfbc96a00064d20d P 59e579024b004c1ebea39d11c0ce6461 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50709fadaa9e44cf830396ddbb1b72a4" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:25:08.787083 19634 consensus_queue.cc:1035] T a70d2c2f94ca4bb5bfbc96a00064d20d P 59e579024b004c1ebea39d11c0ce6461 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e507acf6b90b45269c0f77c976e485ca" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:25:08.788524 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.788199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46815 (local address 127.11.229.2:38103)
0923 17:25:08.788370 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:08.788437 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260923 17:25:08.791194 19637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.790885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46815 (local address 127.11.229.1:46849)
0923 17:25:08.791041 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:08.791108 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260923 17:25:08.791618 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.791168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46815 (local address 127.11.229.2:34733)
0923 17:25:08.791494 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:08.791548 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261}
I20260923 17:25:08.791836 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.791494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46815 (local address 127.11.229.3:50249)
0923 17:25:08.791709 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:08.791766 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
I20260923 17:25:08.793846 19647 mvcc.cc:205] Tried to move back new op lower bound from 7332594928782729216 to 7332594928443088896. Current Snapshot: MvccSnapshot[applied={T|T < 7332594928782729216}]
I20260923 17:25:08.794299 19637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.794042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46815 (local address 127.11.229.1:47399)
0923 17:25:08.794175 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:08.794230 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260923 17:25:08.796090 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.795859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:46815 (local address 127.11.229.3:33405)
0923 17:25:08.795973 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:08.796024 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260923 17:25:08.801334 19082 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:25:08.801517 19082 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:25:08.805613 19082 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b163f7eb24d54419aa3e1337a7c013ae: Sending DeleteTablet for 3 replicas of tablet a70d2c2f94ca4bb5bfbc96a00064d20d
I20260923 17:25:08.806305 19194 tablet_service.cc:1514] Processing DeleteTablet for tablet a70d2c2f94ca4bb5bfbc96a00064d20d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:25:08 UTC) from {username='oryx', principal='oryx/127.11.229.60@KRBTEST.COM'} at 127.0.0.1:41228
I20260923 17:25:08.806475 19467 tablet_service.cc:1514] Processing DeleteTablet for tablet a70d2c2f94ca4bb5bfbc96a00064d20d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:25:08 UTC) from {username='oryx', principal='oryx/127.11.229.60@KRBTEST.COM'} at 127.0.0.1:38652
I20260923 17:25:08.806648 19331 tablet_service.cc:1514] Processing DeleteTablet for tablet a70d2c2f94ca4bb5bfbc96a00064d20d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:25:08 UTC) from {username='oryx', principal='oryx/127.11.229.60@KRBTEST.COM'} at 127.0.0.1:54494
I20260923 17:25:08.806715 19654 tablet_replica.cc:330] T a70d2c2f94ca4bb5bfbc96a00064d20d P e507acf6b90b45269c0f77c976e485ca: stopping tablet replica
I20260923 17:25:08.806866 19654 raft_consensus.cc:2229] T a70d2c2f94ca4bb5bfbc96a00064d20d P e507acf6b90b45269c0f77c976e485ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:25:08.807024 19654 raft_consensus.cc:2258] T a70d2c2f94ca4bb5bfbc96a00064d20d P e507acf6b90b45269c0f77c976e485ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:25:08.807133 19656 tablet_replica.cc:330] T a70d2c2f94ca4bb5bfbc96a00064d20d P 50709fadaa9e44cf830396ddbb1b72a4: stopping tablet replica
I20260923 17:25:08.807333 19656 raft_consensus.cc:2229] T a70d2c2f94ca4bb5bfbc96a00064d20d P 50709fadaa9e44cf830396ddbb1b72a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:25:08.807399 19655 tablet_replica.cc:330] T a70d2c2f94ca4bb5bfbc96a00064d20d P 59e579024b004c1ebea39d11c0ce6461: stopping tablet replica
I20260923 17:25:08.807516 19655 raft_consensus.cc:2229] T a70d2c2f94ca4bb5bfbc96a00064d20d P 59e579024b004c1ebea39d11c0ce6461 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:25:08.807613 19656 raft_consensus.cc:2258] T a70d2c2f94ca4bb5bfbc96a00064d20d P 50709fadaa9e44cf830396ddbb1b72a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:25:08.807677 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 19042
I20260923 17:25:08.807837 19655 raft_consensus.cc:2258] T a70d2c2f94ca4bb5bfbc96a00064d20d P 59e579024b004c1ebea39d11c0ce6461 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:25:08.808933 19656 ts_tablet_manager.cc:1905] T a70d2c2f94ca4bb5bfbc96a00064d20d P 50709fadaa9e44cf830396ddbb1b72a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:25:08.808933 19655 ts_tablet_manager.cc:1905] T a70d2c2f94ca4bb5bfbc96a00064d20d P 59e579024b004c1ebea39d11c0ce6461: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:25:08.811590 19655 ts_tablet_manager.cc:1918] T a70d2c2f94ca4bb5bfbc96a00064d20d P 59e579024b004c1ebea39d11c0ce6461: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:25:08.811590 19656 ts_tablet_manager.cc:1918] T a70d2c2f94ca4bb5bfbc96a00064d20d P 50709fadaa9e44cf830396ddbb1b72a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:25:08.811677 19655 log.cc:1196] T a70d2c2f94ca4bb5bfbc96a00064d20d P 59e579024b004c1ebea39d11c0ce6461: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-2/wal/wals/a70d2c2f94ca4bb5bfbc96a00064d20d
I20260923 17:25:08.811817 19656 log.cc:1196] T a70d2c2f94ca4bb5bfbc96a00064d20d P 50709fadaa9e44cf830396ddbb1b72a4: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184271597484-12180-0/minicluster-data/ts-1/wal/wals/a70d2c2f94ca4bb5bfbc96a00064d20d
I20260923 17:25:08.812007 19655 ts_tablet_manager.cc:1939] T a70d2c2f94ca4bb5bfbc96a00064d20d P 59e579024b004c1ebea39d11c0ce6461: Deleting consensus metadata
I20260923 17:25:08.812229 19656 ts_tablet_manager.cc:1939] T a70d2c2f94ca4bb5bfbc96a00064d20d P 50709fadaa9e44cf830396ddbb1b72a4: Deleting consensus metadata
I20260923 17:25:08.812813 19068 catalog_manager.cc:4841] TS 59e579024b004c1ebea39d11c0ce6461 (127.11.229.3:35491): tablet a70d2c2f94ca4bb5bfbc96a00064d20d (table test-table [id=03bd9c04a68f49eab5305af6dff78ee7]) successfully deleted
I20260923 17:25:08.813077 19066 catalog_manager.cc:4841] TS 50709fadaa9e44cf830396ddbb1b72a4 (127.11.229.2:36405): tablet a70d2c2f94ca4bb5bfbc96a00064d20d (table test-table [id=03bd9c04a68f49eab5305af6dff78ee7]) successfully deleted
I20260923 17:25:08.821031 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 19268
I20260923 17:25:08.821367 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:08.820924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:36025 (local address 127.0.0.1:41242)
0923 17:25:08.821139 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:08.821219 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.1:36025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260923 17:25:08.821897 19065 catalog_manager.cc:4584] TS e507acf6b90b45269c0f77c976e485ca (127.11.229.1:36025): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a70d2c2f94ca4bb5bfbc96a00064d20d: Network error: Client connection negotiation failed: client connection to 127.11.229.1:36025: connect: Connection refused (error 111)
I20260923 17:25:08.828507 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 19404
I20260923 17:25:08.834636 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 18761
I20260923 17:25:08.840708 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 18868
2026-09-23T17:25:08Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5665 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:25:08 dist-test-slave-ptl9 krb5kdc[19659](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:25:08 dist-test-slave-ptl9 krb5kdc[19659](info): set up 2 sockets
Sep 23 17:25:08 dist-test-slave-ptl9 krb5kdc[19659](info): commencing operation
krb5kdc: starting...
W20260923 17:25:09.886300 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:25:09 dist-test-slave-ptl9 krb5kdc[19659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:25:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:25:09Z Disabled control of system clock
WARNING: no policy specified for oryx/127.11.229.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.229.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.229.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.62.keytab.
Entry for principal oryx/127.11.229.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.62.keytab.
I20260923 17:25:10.012732 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:42063
--webserver_interface=127.11.229.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.62.keytab
--principal=oryx/127.11.229.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:36675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:42063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:25:10.125999 19672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:10.126332 19672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:10.126435 19672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:10.130616 19672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:25:10.130705 19672 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 17:25:10.130738 19672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:10.130801 19672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:10.130857 19672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:25:10.130894 19672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:25:10.136430 19672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:36675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:42063
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.62.keytab
--principal=oryx/127.11.229.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:42063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:25:10.137622 19672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:10.138687 19672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:10.147557 19677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:10.147574 19680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:10.147591 19678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:10.147823 19672 server_base.cc:1034] running on GCE node
I20260923 17:25:10.148579 19672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:10.149761 19672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:10.150969 19672 hybrid_clock.cc:648] HybridClock initialized: now 1790184310150938 us; error 53 us; skew 500 ppm
Sep 23 17:25:10 dist-test-slave-ptl9 krb5kdc[19659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184310, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:10.158905 19672 init.cc:377] Logged in from keytab as oryx/127.11.229.62@KRBTEST.COM (short username oryx)
I20260923 17:25:10.159963 19672 webserver.cc:460] Webserver started at http://127.11.229.62:41929/ using document root <none> and password file <none>
I20260923 17:25:10.160604 19672 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:10.160698 19672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:10.160949 19672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:25:10.163563 19672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "b6dbd2dab05c43ae96c5126bcbfbf2b4"
format_stamp: "Formatted at 2026-09-23 17:25:10 on dist-test-slave-ptl9"
server_key: "382a0693a27def285953168a28afebef"
server_key_iv: "b98f234e08cb551d1e0263079d79458d"
server_key_version: "encryptionkey@0"
I20260923 17:25:10.164160 19672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "b6dbd2dab05c43ae96c5126bcbfbf2b4"
format_stamp: "Formatted at 2026-09-23 17:25:10 on dist-test-slave-ptl9"
server_key: "382a0693a27def285953168a28afebef"
server_key_iv: "b98f234e08cb551d1e0263079d79458d"
server_key_version: "encryptionkey@0"
I20260923 17:25:10.168228 19672 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260923 17:25:10.170799 19687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:10.172089 19672 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:25:10.172268 19672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "b6dbd2dab05c43ae96c5126bcbfbf2b4"
format_stamp: "Formatted at 2026-09-23 17:25:10 on dist-test-slave-ptl9"
server_key: "382a0693a27def285953168a28afebef"
server_key_iv: "b98f234e08cb551d1e0263079d79458d"
server_key_version: "encryptionkey@0"
I20260923 17:25:10.172390 19672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:10.205382 19672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:10.208379 19672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:10.208606 19672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:10.216979 19672 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:42063
I20260923 17:25:10.216996 19749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:42063 every 8 connection(s)
I20260923 17:25:10.218043 19672 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:25:10.219713 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 19672
I20260923 17:25:10.219848 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:25:10.220185 12180 external_mini_cluster.cc:1428] Setting key 12002cb98857c50273793ca00285c1c5
I20260923 17:25:10.221426 19750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:10.227257 19750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6dbd2dab05c43ae96c5126bcbfbf2b4: Bootstrap starting.
Sep 23 17:25:10 dist-test-slave-ptl9 krb5kdc[19659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM
I20260923 17:25:10.229799 19750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6dbd2dab05c43ae96c5126bcbfbf2b4: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:10.231149 19750 log.cc:824] T 00000000000000000000000000000000 P b6dbd2dab05c43ae96c5126bcbfbf2b4: Log is configured to *not* fsync() on all Append() calls
I20260923 17:25:10.233541 19750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6dbd2dab05c43ae96c5126bcbfbf2b4: No bootstrap required, opened a new log
I20260923 17:25:10.234455 19753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:10.221753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41424 (local address 127.11.229.62:42063)
0923 17:25:10.222230 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:10.222246 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:10.222282 (+ 36us) server_negotiation.cc:407] Connection header received
0923 17:25:10.222937 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:10.222953 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:10.223291 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:10.223644 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:10.224475 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:10.225561 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:10.226420 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:10.226910 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:10.227109 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:10.229655 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:10.229661 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:10.229674 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:10.229702 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:10.231549 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:10.232038 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:10.232044 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:10.232048 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:10.232169 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:10.232459 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:10.232461 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:10.232462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:10.232854 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:10.233016 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:10.233386 (+ 370us) server_negotiation.cc:299] Negotiation successful
0923 17:25:10.233719 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":142,"threads_started":1}
I20260923 17:25:10.236635 19750 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6dbd2dab05c43ae96c5126bcbfbf2b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6dbd2dab05c43ae96c5126bcbfbf2b4" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 42063 } }
I20260923 17:25:10.236903 19750 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6dbd2dab05c43ae96c5126bcbfbf2b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:10.236985 19750 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6dbd2dab05c43ae96c5126bcbfbf2b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6dbd2dab05c43ae96c5126bcbfbf2b4, State: Initialized, Role: FOLLOWER
I20260923 17:25:10.237469 19750 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6dbd2dab05c43ae96c5126bcbfbf2b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6dbd2dab05c43ae96c5126bcbfbf2b4" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 42063 } }
I20260923 17:25:10.237632 19750 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6dbd2dab05c43ae96c5126bcbfbf2b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:25:10.237725 19750 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6dbd2dab05c43ae96c5126bcbfbf2b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:25:10.237823 19750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6dbd2dab05c43ae96c5126bcbfbf2b4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:10.238722 19750 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6dbd2dab05c43ae96c5126bcbfbf2b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6dbd2dab05c43ae96c5126bcbfbf2b4" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 42063 } }
I20260923 17:25:10.239074 19750 leader_election.cc:302] T 00000000000000000000000000000000 P b6dbd2dab05c43ae96c5126bcbfbf2b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6dbd2dab05c43ae96c5126bcbfbf2b4; no voters:
I20260923 17:25:10.239449 19750 leader_election.cc:288] T 00000000000000000000000000000000 P b6dbd2dab05c43ae96c5126bcbfbf2b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:25:10.239568 19755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6dbd2dab05c43ae96c5126bcbfbf2b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:25:10.239828 19755 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6dbd2dab05c43ae96c5126bcbfbf2b4 [term 1 LEADER]: Becoming Leader. State: Replica: b6dbd2dab05c43ae96c5126bcbfbf2b4, State: Running, Role: LEADER
I20260923 17:25:10.240163 19755 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6dbd2dab05c43ae96c5126bcbfbf2b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6dbd2dab05c43ae96c5126bcbfbf2b4" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 42063 } }
I20260923 17:25:10.240706 19750 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6dbd2dab05c43ae96c5126bcbfbf2b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:25:10.242000 19757 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6dbd2dab05c43ae96c5126bcbfbf2b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6dbd2dab05c43ae96c5126bcbfbf2b4. Latest consensus state: current_term: 1 leader_uuid: "b6dbd2dab05c43ae96c5126bcbfbf2b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6dbd2dab05c43ae96c5126bcbfbf2b4" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 42063 } } }
I20260923 17:25:10.242007 19756 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6dbd2dab05c43ae96c5126bcbfbf2b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6dbd2dab05c43ae96c5126bcbfbf2b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6dbd2dab05c43ae96c5126bcbfbf2b4" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 42063 } } }
I20260923 17:25:10.242141 19757 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6dbd2dab05c43ae96c5126bcbfbf2b4 [sys.catalog]: This master's current role is: LEADER
I20260923 17:25:10.242144 19756 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6dbd2dab05c43ae96c5126bcbfbf2b4 [sys.catalog]: This master's current role is: LEADER
I20260923 17:25:10.242911 19764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:25:10.245710 19764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:25:10.251314 19764 catalog_manager.cc:1347] Generated new cluster ID: b3303fcef8224d4ca1927bef4520e074
I20260923 17:25:10.251384 19764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:25:10.263734 19764 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:25:10.265008 19764 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:25:10.270102 19764 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6dbd2dab05c43ae96c5126bcbfbf2b4: Generated new TSK 0
I20260923 17:25:10.270941 19764 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.11.229.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.229.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.229.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.1.keytab.
Entry for principal oryx/127.11.229.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.1.keytab.
I20260923 17:25:10.304141 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.1:0
--local_ip_for_outbound_sockets=127.11.229.1
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:42063
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.1.keytab
--principal=oryx/127.11.229.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:36675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:25:10.421715 19776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:10.421974 19776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:10.422021 19776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:10.425946 19776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:10.426004 19776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:10.426054 19776 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:25:10.426087 19776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.1
I20260923 17:25:10.431448 19776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:36675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.1.keytab
--principal=oryx/127.11.229.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.229.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.229.62:42063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.1
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:25:10.432556 19776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:10.433678 19776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:10.444088 19782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:10.444139 19784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:10.444128 19781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:10.444275 19776 server_base.cc:1034] running on GCE node
I20260923 17:25:10.444849 19776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:10.445623 19776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:10.446805 19776 hybrid_clock.cc:648] HybridClock initialized: now 1790184310446775 us; error 40 us; skew 500 ppm
Sep 23 17:25:10 dist-test-slave-ptl9 krb5kdc[19659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184310, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:10.449739 19776 init.cc:377] Logged in from keytab as oryx/127.11.229.1@KRBTEST.COM (short username oryx)
I20260923 17:25:10.450757 19776 webserver.cc:460] Webserver started at http://127.11.229.1:39121/ using document root <none> and password file <none>
I20260923 17:25:10.451373 19776 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:10.451437 19776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:10.451617 19776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:25:10.453845 19776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-0/data/instance:
uuid: "494144bdd0314b94aa649bab67334db3"
format_stamp: "Formatted at 2026-09-23 17:25:10 on dist-test-slave-ptl9"
server_key: "5bd4ed3492450fad9e1b85b99f503773"
server_key_iv: "80e49c236c61c247fb061e03ea747079"
server_key_version: "encryptionkey@0"
I20260923 17:25:10.454380 19776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance:
uuid: "494144bdd0314b94aa649bab67334db3"
format_stamp: "Formatted at 2026-09-23 17:25:10 on dist-test-slave-ptl9"
server_key: "5bd4ed3492450fad9e1b85b99f503773"
server_key_iv: "80e49c236c61c247fb061e03ea747079"
server_key_version: "encryptionkey@0"
I20260923 17:25:10.458269 19776 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:25:10.460852 19791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:10.462751 19776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:25:10.462934 19776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-0/wal
uuid: "494144bdd0314b94aa649bab67334db3"
format_stamp: "Formatted at 2026-09-23 17:25:10 on dist-test-slave-ptl9"
server_key: "5bd4ed3492450fad9e1b85b99f503773"
server_key_iv: "80e49c236c61c247fb061e03ea747079"
server_key_version: "encryptionkey@0"
I20260923 17:25:10.463063 19776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:10.489025 19776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:10.492722 19776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:10.492955 19776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:10.494589 19776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:25:10.494668 19776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:10.494732 19776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:25:10.494784 19776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:25:10 dist-test-slave-ptl9 krb5kdc[19659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184310, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM
I20260923 17:25:10.505815 19776 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.1:34095
I20260923 17:25:10.505888 19906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.1:34095 every 8 connection(s)
I20260923 17:25:10.507056 19776 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
I20260923 17:25:10.510079 19753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:10.495744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:58555 (local address 127.11.229.62:42063)
0923 17:25:10.495907 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:10.495922 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:10.496860 (+ 938us) server_negotiation.cc:407] Connection header received
0923 17:25:10.498314 (+ 1454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:10.498322 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:10.498395 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:10.498502 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:10.499934 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:10.500870 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:10.502506 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:10.502879 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:10.503005 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:10.505889 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:10.505893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:10.505896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:10.505926 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:10.507606 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:10.508259 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:10.508263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:10.508265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:10.508366 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:10.508732 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:10.508736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:10.508738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:10.508940 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:10.509063 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:10.509755 (+ 692us) server_negotiation.cc:299] Negotiation successful
0923 17:25:10.509893 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 17:25:10.510871 19800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:10.496195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:42063 (local address 127.11.229.1:58555)
0923 17:25:10.496683 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:10.496736 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:10.498020 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:10.498689 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:10.498696 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:10.499246 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:10.499783 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:10.499798 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:10.501004 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:10.501009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:10.502355 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:10.502366 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:10.503052 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:10.503057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:10.503250 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:10.503901 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:10.503932 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:10.505676 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:10.507750 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:10.507756 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:10.507773 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:10.508109 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:10.508447 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:10.508451 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:10.508453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:10.508604 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:10.509058 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:10.509065 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:10.509458 (+ 393us) client_negotiation.cc:769] Sending connection context
0923 17:25:10.509695 (+ 237us) client_negotiation.cc:240] Negotiation successful
0923 17:25:10.510031 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":152,"threads_started":1}
I20260923 17:25:10.511082 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 19776
I20260923 17:25:10.511232 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance
I20260923 17:25:10.511595 12180 external_mini_cluster.cc:1428] Setting key 71fec71eb86f2587b431af93b57a1d59
I20260923 17:25:10.512312 19907 heartbeater.cc:347] Connected to a master server at 127.11.229.62:42063
I20260923 17:25:10.512490 19907 heartbeater.cc:464] Registering TS with master...
I20260923 17:25:10.513087 19907 heartbeater.cc:511] Master 127.11.229.62:42063 requested a full tablet report, sending...
I20260923 17:25:10.514817 19703 ts_manager.cc:194] Registered new tserver with Master: 494144bdd0314b94aa649bab67334db3 (127.11.229.1:34095)
I20260923 17:25:10.516268 19703 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.229.1@KRBTEST.COM'} at 127.11.229.1:58555
WARNING: no policy specified for oryx/127.11.229.2@KRBTEST.COM; defaulting to no policy
I20260923 17:25:10.523015 19914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:10.515078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:42063 (local address 127.11.229.1:44973)
0923 17:25:10.515398 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:10.515415 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:10.515523 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:10.515783 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:10.515787 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:10.516022 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:10.516180 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:10.516188 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:10.517053 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:10.517056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:10.517970 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:10.517980 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:10.518656 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:10.518660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:10.518756 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:10.519313 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:10.519333 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:10.519668 (+ 335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:10.521623 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:10.521626 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:10.521628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:10.521826 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:10.522120 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:10.522125 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:10.522127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:10.522215 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:10.522592 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:10.522594 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:10.522672 (+ 78us) client_negotiation.cc:769] Sending connection context
0923 17:25:10.522761 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 17:25:10.522865 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":1,"thread_start_us":104,"threads_started":1}
I20260923 17:25:10.523022 19753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:10.514875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:44973 (local address 127.11.229.62:42063)
0923 17:25:10.514988 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:10.514992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:10.515513 (+ 521us) server_negotiation.cc:407] Connection header received
0923 17:25:10.515613 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:10.515616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:10.515660 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:10.515742 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:10.516302 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:10.516924 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:10.518111 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:10.518414 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:10.518491 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:10.519808 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:10.519811 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:10.519814 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:10.519837 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:10.521512 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:10.521928 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:10.521932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:10.521934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:10.522019 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:10.522315 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:10.522318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:10.522319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:10.522496 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:10.522579 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:10.522823 (+ 244us) server_negotiation.cc:299] Negotiation successful
0923 17:25:10.522915 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.229.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.229.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.2.keytab.
Entry for principal oryx/127.11.229.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.2.keytab.
I20260923 17:25:10.540082 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.2:0
--local_ip_for_outbound_sockets=127.11.229.2
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:42063
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.2.keytab
--principal=oryx/127.11.229.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:36675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:25:10.654706 19916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:10.654963 19916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:10.655007 19916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:10.659087 19916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:10.659155 19916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:10.659195 19916 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:25:10.659228 19916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.2
I20260923 17:25:10.664558 19916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:36675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.2.keytab
--principal=oryx/127.11.229.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.229.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.229.62:42063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.2
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:25:10.665699 19916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:10.666780 19916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:10.676966 19924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:10.677019 19922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:10.676973 19921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:10.677299 19916 server_base.cc:1034] running on GCE node
I20260923 17:25:10.677719 19916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:10.678380 19916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:10.679575 19916 hybrid_clock.cc:648] HybridClock initialized: now 1790184310679560 us; error 37 us; skew 500 ppm
Sep 23 17:25:10 dist-test-slave-ptl9 krb5kdc[19659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184310, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:10.682442 19916 init.cc:377] Logged in from keytab as oryx/127.11.229.2@KRBTEST.COM (short username oryx)
I20260923 17:25:10.683285 19916 webserver.cc:460] Webserver started at http://127.11.229.2:42391/ using document root <none> and password file <none>
I20260923 17:25:10.683809 19916 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:10.683864 19916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:10.684032 19916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:25:10.686210 19916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-1/data/instance:
uuid: "72e833a06f154110a6a8ac4625fac572"
format_stamp: "Formatted at 2026-09-23 17:25:10 on dist-test-slave-ptl9"
server_key: "cbd7b7bc3244f26e0c20b08d4e642b9f"
server_key_iv: "40a3954664c42a5f27ab53a7dfdf70ba"
server_key_version: "encryptionkey@0"
I20260923 17:25:10.686805 19916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance:
uuid: "72e833a06f154110a6a8ac4625fac572"
format_stamp: "Formatted at 2026-09-23 17:25:10 on dist-test-slave-ptl9"
server_key: "cbd7b7bc3244f26e0c20b08d4e642b9f"
server_key_iv: "40a3954664c42a5f27ab53a7dfdf70ba"
server_key_version: "encryptionkey@0"
I20260923 17:25:10.690666 19916 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:25:10.693038 19931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:10.694267 19916 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:25:10.694389 19916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-1/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-1/wal
uuid: "72e833a06f154110a6a8ac4625fac572"
format_stamp: "Formatted at 2026-09-23 17:25:10 on dist-test-slave-ptl9"
server_key: "cbd7b7bc3244f26e0c20b08d4e642b9f"
server_key_iv: "40a3954664c42a5f27ab53a7dfdf70ba"
server_key_version: "encryptionkey@0"
I20260923 17:25:10.694496 19916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:10.711486 19916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:10.715030 19916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:10.715238 19916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:10.716780 19916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:25:10.716835 19916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:10.716904 19916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:25:10.716938 19916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:25:10 dist-test-slave-ptl9 krb5kdc[19659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184310, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM
I20260923 17:25:10.728245 19916 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.2:33753
I20260923 17:25:10.729099 20046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.2:33753 every 8 connection(s)
I20260923 17:25:10.729450 19916 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
I20260923 17:25:10.731060 19753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:10.717681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:44027 (local address 127.11.229.62:42063)
0923 17:25:10.717838 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:10.717843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:10.718819 (+ 976us) server_negotiation.cc:407] Connection header received
0923 17:25:10.720243 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:10.720250 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:10.720318 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:10.720437 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:10.721865 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:10.722423 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:10.723727 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:10.724025 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:10.724153 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:10.727185 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:10.727189 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:10.727192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:10.727226 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:10.728899 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:10.729503 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:10.729507 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:10.729508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:10.729601 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:10.729980 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:10.729984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:10.729985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:10.730215 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:10.730327 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:10.730827 (+ 500us) server_negotiation.cc:299] Negotiation successful
0923 17:25:10.730938 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 17:25:10.731853 19940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:10.718123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:42063 (local address 127.11.229.2:44027)
0923 17:25:10.718636 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:10.718700 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:10.719923 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:10.720636 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:10.720643 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:10.721109 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:10.721708 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:10.721722 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:10.722560 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:10.722564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:10.723593 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:10.723603 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:10.724216 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:10.724221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:10.724383 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:10.725034 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:10.725051 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:10.726997 (+ 1946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:10.729012 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:10.729017 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:10.729029 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:10.729387 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:10.729689 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:10.729692 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:10.729693 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:10.729856 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:10.730319 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:10.730326 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:10.730592 (+ 266us) client_negotiation.cc:769] Sending connection context
0923 17:25:10.730745 (+ 153us) client_negotiation.cc:240] Negotiation successful
0923 17:25:10.730997 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":112,"threads_started":1}
I20260923 17:25:10.733251 20047 heartbeater.cc:347] Connected to a master server at 127.11.229.62:42063
I20260923 17:25:10.733454 20047 heartbeater.cc:464] Registering TS with master...
I20260923 17:25:10.733948 20047 heartbeater.cc:511] Master 127.11.229.62:42063 requested a full tablet report, sending...
I20260923 17:25:10.735033 19703 ts_manager.cc:194] Registered new tserver with Master: 72e833a06f154110a6a8ac4625fac572 (127.11.229.2:33753)
I20260923 17:25:10.735805 19703 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.229.2@KRBTEST.COM'} at 127.11.229.2:44027
I20260923 17:25:10.737026 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 19916
I20260923 17:25:10.737149 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance
I20260923 17:25:10.737471 12180 external_mini_cluster.cc:1428] Setting key e1fd9d96186ed844260a9aa7644e01b5
I20260923 17:25:10.743321 19753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:10.735591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:42269 (local address 127.11.229.62:42063)
0923 17:25:10.735727 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:10.735731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:10.735957 (+ 226us) server_negotiation.cc:407] Connection header received
0923 17:25:10.736073 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:10.736076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:10.736122 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:10.736205 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:10.736670 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:10.737219 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:10.738221 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:10.738453 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:10.738565 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:10.739637 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:10.739641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:10.739644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:10.739668 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:10.741371 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:10.741991 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:10.741993 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:10.741994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:10.742064 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:10.742360 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:10.742363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:10.742364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:10.742634 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:10.742743 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:10.743037 (+ 294us) server_negotiation.cc:299] Negotiation successful
0923 17:25:10.743145 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 17:25:10.743330 20053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:10.735516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:42063 (local address 127.11.229.2:42269)
0923 17:25:10.735883 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:10.735899 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:10.735997 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:10.736222 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:10.736224 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:10.736438 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:10.736553 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:10.736561 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:10.737352 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:10.737358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:10.738104 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:10.738114 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:10.738567 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:10.738571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:10.738670 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:10.739168 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:10.739179 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:10.739501 (+ 322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:10.741517 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:10.741520 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:10.741522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:10.741848 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:10.742158 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:10.742163 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:10.742167 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:10.742257 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:10.742732 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:10.742735 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:10.742837 (+ 102us) client_negotiation.cc:769] Sending connection context
0923 17:25:10.742946 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 17:25:10.743096 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for oryx/127.11.229.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.229.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.229.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.3.keytab.
Entry for principal oryx/127.11.229.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.3.keytab.
I20260923 17:25:10.765846 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.3:0
--local_ip_for_outbound_sockets=127.11.229.3
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:42063
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.3.keytab
--principal=oryx/127.11.229.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:36675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:25:10.878762 20056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:10.879024 20056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:10.879070 20056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:10.883056 20056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:10.883111 20056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:10.883149 20056 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:25:10.883183 20056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.3
I20260923 17:25:10.888334 20056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:36675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.3.keytab
--principal=oryx/127.11.229.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.229.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.229.62:42063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.3
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:25:10.889411 20056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:10.890419 20056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:10.900322 20061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:10.900322 20062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:10.900329 20064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:10.901006 20056 server_base.cc:1034] running on GCE node
I20260923 17:25:10.901487 20056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:10.902134 20056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:10.903324 20056 hybrid_clock.cc:648] HybridClock initialized: now 1790184310903292 us; error 48 us; skew 500 ppm
Sep 23 17:25:10 dist-test-slave-ptl9 krb5kdc[19659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184310, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:10.906090 20056 init.cc:377] Logged in from keytab as oryx/127.11.229.3@KRBTEST.COM (short username oryx)
I20260923 17:25:10.906963 20056 webserver.cc:460] Webserver started at http://127.11.229.3:40097/ using document root <none> and password file <none>
I20260923 17:25:10.907544 20056 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:10.907636 20056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:10.907876 20056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:25:10.910019 20056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-2/data/instance:
uuid: "d9a66fcfdb744bd6bd7db41019abd08f"
format_stamp: "Formatted at 2026-09-23 17:25:10 on dist-test-slave-ptl9"
server_key: "c0e7da95a64e7691c71cc357368f95c0"
server_key_iv: "9cf0508babf035b1d894ec43b5d6726d"
server_key_version: "encryptionkey@0"
I20260923 17:25:10.910624 20056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance:
uuid: "d9a66fcfdb744bd6bd7db41019abd08f"
format_stamp: "Formatted at 2026-09-23 17:25:10 on dist-test-slave-ptl9"
server_key: "c0e7da95a64e7691c71cc357368f95c0"
server_key_iv: "9cf0508babf035b1d894ec43b5d6726d"
server_key_version: "encryptionkey@0"
I20260923 17:25:10.914665 20056 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:25:10.917136 20071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:10.918367 20056 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 17:25:10.918504 20056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-2/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-2/wal
uuid: "d9a66fcfdb744bd6bd7db41019abd08f"
format_stamp: "Formatted at 2026-09-23 17:25:10 on dist-test-slave-ptl9"
server_key: "c0e7da95a64e7691c71cc357368f95c0"
server_key_iv: "9cf0508babf035b1d894ec43b5d6726d"
server_key_version: "encryptionkey@0"
I20260923 17:25:10.918634 20056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:10.934211 20056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:10.937225 20056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:10.937484 20056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:10.938912 20056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:25:10.938987 20056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:10.939054 20056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:25:10.939101 20056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:25:10 dist-test-slave-ptl9 krb5kdc[19659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184310, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.3@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM
I20260923 17:25:10.950302 20056 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.3:41689
I20260923 17:25:10.950325 20186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.3:41689 every 8 connection(s)
I20260923 17:25:10.951272 20056 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
I20260923 17:25:10.952301 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 20056
I20260923 17:25:10.952442 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance
I20260923 17:25:10.952771 12180 external_mini_cluster.cc:1428] Setting key eacdf0bf8c645cbbed36e97d1ca5bfea
I20260923 17:25:10.954020 19753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:10.939864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:44785 (local address 127.11.229.62:42063)
0923 17:25:10.939996 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:10.940000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:10.941214 (+ 1214us) server_negotiation.cc:407] Connection header received
0923 17:25:10.942668 (+ 1454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:10.942676 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:10.942776 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:10.942882 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:10.944442 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:10.945432 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:10.946753 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:10.947152 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:10.947280 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:10.949792 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:10.949796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:10.949799 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:10.949829 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:10.951520 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:10.952111 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:10.952115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:10.952117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:10.952213 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:10.952631 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:10.952634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:10.952635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:10.952873 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:10.952992 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:10.953737 (+ 745us) server_negotiation.cc:299] Negotiation successful
0923 17:25:10.953857 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 17:25:10.954839 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:10.940452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:42063 (local address 127.11.229.3:44785)
0923 17:25:10.941061 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:10.941104 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:10.942384 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:10.943076 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:10.943087 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:10.943537 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:10.944259 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:10.944279 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:10.945567 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:10.945570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:10.946641 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:10.946651 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:10.947321 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:10.947324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:10.947469 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:10.948105 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:10.948132 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:10.949604 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:10.951638 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:10.951644 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:10.951660 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:10.951982 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:10.952328 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:10.952331 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:10.952334 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:10.952530 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:10.953017 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:10.953024 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:10.953343 (+ 319us) client_negotiation.cc:769] Sending connection context
0923 17:25:10.953556 (+ 213us) client_negotiation.cc:240] Negotiation successful
0923 17:25:10.953876 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":144,"threads_started":1}
I20260923 17:25:10.955976 20187 heartbeater.cc:347] Connected to a master server at 127.11.229.62:42063
I20260923 17:25:10.956147 20187 heartbeater.cc:464] Registering TS with master...
I20260923 17:25:10.956736 20187 heartbeater.cc:511] Master 127.11.229.62:42063 requested a full tablet report, sending...
I20260923 17:25:10.957796 19704 ts_manager.cc:194] Registered new tserver with Master: d9a66fcfdb744bd6bd7db41019abd08f (127.11.229.3:41689)
I20260923 17:25:10.958602 19704 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.229.3@KRBTEST.COM'} at 127.11.229.3:44785
I20260923 17:25:10.964792 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:10.958052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:42063 (local address 127.11.229.3:57121)
0923 17:25:10.958340 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:10.958354 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:10.958446 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:10.958678 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:10.958683 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:10.958888 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:10.958992 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:10.958998 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:10.959757 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:10.959760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:10.960575 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:10.960584 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:10.961004 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:10.961008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:10.961081 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:10.961616 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:10.961627 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:10.961926 (+ 299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:10.963554 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:10.963557 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:10.963558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:10.963727 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:10.963960 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:10.963962 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:10.963963 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:10.964033 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:10.964349 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:10.964351 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:10.964447 (+ 96us) client_negotiation.cc:769] Sending connection context
0923 17:25:10.964555 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 17:25:10.964657 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":107,"threads_started":1}
I20260923 17:25:10.964809 19753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:10.958102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:57121 (local address 127.11.229.62:42063)
0923 17:25:10.958239 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:10.958243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:10.958426 (+ 183us) server_negotiation.cc:407] Connection header received
0923 17:25:10.958531 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:10.958533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:10.958569 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:10.958632 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:10.959101 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:10.959625 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:10.960678 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:10.960884 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:10.960978 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:10.962047 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:10.962049 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:10.962051 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:10.962073 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:10.963463 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:10.963809 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:10.963811 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:10.963813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:10.963883 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:10.964107 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:10.964109 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:10.964110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:10.964271 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:10.964345 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:10.964616 (+ 271us) server_negotiation.cc:299] Negotiation successful
0923 17:25:10.964703 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 17:25:10.967252 12180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 17:25:10 dist-test-slave-ptl9 krb5kdc[19659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184310, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 17:25:10 dist-test-slave-ptl9 krb5kdc[19659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:10 dist-test-slave-ptl9 krb5kdc[19659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
W20260923 17:25:10.990557 19753 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:10.984460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41432 (local address 127.11.229.62:42063)
0923 17:25:10.984601 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:10.984605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:10.984616 (+ 11us) server_negotiation.cc:407] Connection header received
0923 17:25:10.984656 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:10.984658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:10.984695 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:10.984758 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:10.985542 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:10.986070 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:10.986921 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:10.987145 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:10.987227 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:10.990434 (+ 3207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41432: BlockingRecv error: recv got EOF from 127.0.0.1:41432 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 23 17:25:10 dist-test-slave-ptl9 krb5kdc[19659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184310, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM
I20260923 17:25:11.000061 19753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:10.992037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41448 (local address 127.11.229.62:42063)
0923 17:25:10.992185 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:10.992188 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:10.992242 (+ 54us) server_negotiation.cc:407] Connection header received
0923 17:25:10.992338 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:10.992340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:10.992377 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:10.992444 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:10.992996 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:10.993552 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:10.994314 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:10.994517 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:10.994589 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:10.996763 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:10.996767 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:10.996769 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:10.996792 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:10.998340 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:10.998858 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:10.998861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:10.998862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:10.998940 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:10.999242 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:10.999245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:10.999246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:10.999422 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:10.999526 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:10.999753 (+ 227us) server_negotiation.cc:299] Negotiation successful
0923 17:25:10.999849 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 17:25:11.003197 19704 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41448:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:25:11.006376 19704 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:25:11.022110 20214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:11.016675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44786 (local address 127.11.229.1:34095)
0923 17:25:11.017133 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:11.017141 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:11.017436 (+ 295us) server_negotiation.cc:407] Connection header received
0923 17:25:11.017728 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:11.017737 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:11.017963 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:11.018092 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:11.018512 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.019387 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.020651 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.021377 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.021487 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:11.021642 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:11.021864 (+ 222us) server_negotiation.cc:299] Negotiation successful
0923 17:25:11.021961 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":93,"threads_started":1}
I20260923 17:25:11.022104 20212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:11.016422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:34095 (local address 127.0.0.1:44786)
0923 17:25:11.017329 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:11.017345 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:11.017538 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:11.018273 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:11.018276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:11.018291 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:11.018380 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:11.018384 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.019513 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.019516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:11.020511 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:11.020527 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.021509 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.021512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:11.021661 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:11.021695 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 17:25:11.021823 (+ 128us) client_negotiation.cc:240] Negotiation successful
0923 17:25:11.021950 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"mutex_wait_us":75,"spinlock_wait_cycles":768,"thread_start_us":95,"threads_started":1}
I20260923 17:25:11.022753 20213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:11.015568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:41689 (local address 127.0.0.1:50534)
0923 17:25:11.016719 (+ 1151us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:11.016733 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:11.016820 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:11.017467 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:11.017474 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:11.017515 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:11.017818 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:11.017826 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.019288 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.019291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:11.021007 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:11.021014 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.022312 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.022316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:11.022417 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:11.022441 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 17:25:11.022515 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 17:25:11.022585 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1083,"mutex_wait_us":801,"spinlock_wait_cycles":7808,"thread_start_us":105,"threads_started":1}
I20260923 17:25:11.023183 20211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:11.015800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50534 (local address 127.11.229.3:41689)
0923 17:25:11.016410 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:11.016422 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:11.016905 (+ 483us) server_negotiation.cc:407] Connection header received
0923 17:25:11.016978 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:11.016982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:11.017122 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:11.017293 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:11.017957 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.019150 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.021134 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.021992 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.022094 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:11.022245 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:11.022916 (+ 671us) server_negotiation.cc:299] Negotiation successful
0923 17:25:11.023024 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":105,"threads_started":1}
I20260923 17:25:11.023902 20209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:11.015568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:33753 (local address 127.0.0.1:57594)
0923 17:25:11.016423 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:11.016476 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:11.016614 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:11.017467 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:11.017474 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:11.017516 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:11.017818 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:11.017826 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.019288 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.019291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:11.020991 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:11.021004 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.023498 (+ 2494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.023506 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:11.023582 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:11.023600 (+ 18us) client_negotiation.cc:769] Sending connection context
0923 17:25:11.023698 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 17:25:11.023775 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":288,"spinlock_wait_cycles":19712,"thread_start_us":64,"threads_started":1}
I20260923 17:25:11.023968 20210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:11.015763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57594 (local address 127.11.229.2:33753)
0923 17:25:11.016090 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:11.016103 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:11.016774 (+ 671us) server_negotiation.cc:407] Connection header received
0923 17:25:11.016869 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:11.016877 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:11.017118 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:11.017293 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:11.017966 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.019150 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.021716 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.022658 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.023230 (+ 572us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:11.023397 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:11.023738 (+ 341us) server_negotiation.cc:299] Negotiation successful
0923 17:25:11.023832 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":131,"threads_started":1}
I20260923 17:25:11.026146 19841 tablet_service.cc:1467] Processing CreateTablet for tablet 00b2ad32c9c04893a426e5c44f84ccf8 (DEFAULT_TABLE table=test-table [id=2384f2864f814925a9814eae635d6afa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:25:11.026292 20121 tablet_service.cc:1467] Processing CreateTablet for tablet 00b2ad32c9c04893a426e5c44f84ccf8 (DEFAULT_TABLE table=test-table [id=2384f2864f814925a9814eae635d6afa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:25:11.026530 19981 tablet_service.cc:1467] Processing CreateTablet for tablet 00b2ad32c9c04893a426e5c44f84ccf8 (DEFAULT_TABLE table=test-table [id=2384f2864f814925a9814eae635d6afa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:25:11.027408 20121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00b2ad32c9c04893a426e5c44f84ccf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:11.027406 19841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00b2ad32c9c04893a426e5c44f84ccf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:11.027406 19981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00b2ad32c9c04893a426e5c44f84ccf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:11.034343 20215 tablet_bootstrap.cc:492] T 00b2ad32c9c04893a426e5c44f84ccf8 P d9a66fcfdb744bd6bd7db41019abd08f: Bootstrap starting.
I20260923 17:25:11.035043 20216 tablet_bootstrap.cc:492] T 00b2ad32c9c04893a426e5c44f84ccf8 P 494144bdd0314b94aa649bab67334db3: Bootstrap starting.
I20260923 17:25:11.036600 20217 tablet_bootstrap.cc:492] T 00b2ad32c9c04893a426e5c44f84ccf8 P 72e833a06f154110a6a8ac4625fac572: Bootstrap starting.
I20260923 17:25:11.037228 20216 tablet_bootstrap.cc:654] T 00b2ad32c9c04893a426e5c44f84ccf8 P 494144bdd0314b94aa649bab67334db3: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:11.037729 20215 tablet_bootstrap.cc:654] T 00b2ad32c9c04893a426e5c44f84ccf8 P d9a66fcfdb744bd6bd7db41019abd08f: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:11.038220 20216 log.cc:824] T 00b2ad32c9c04893a426e5c44f84ccf8 P 494144bdd0314b94aa649bab67334db3: Log is configured to *not* fsync() on all Append() calls
I20260923 17:25:11.038646 20215 log.cc:824] T 00b2ad32c9c04893a426e5c44f84ccf8 P d9a66fcfdb744bd6bd7db41019abd08f: Log is configured to *not* fsync() on all Append() calls
I20260923 17:25:11.038949 20217 tablet_bootstrap.cc:654] T 00b2ad32c9c04893a426e5c44f84ccf8 P 72e833a06f154110a6a8ac4625fac572: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:11.039798 20217 log.cc:824] T 00b2ad32c9c04893a426e5c44f84ccf8 P 72e833a06f154110a6a8ac4625fac572: Log is configured to *not* fsync() on all Append() calls
I20260923 17:25:11.040655 20215 tablet_bootstrap.cc:492] T 00b2ad32c9c04893a426e5c44f84ccf8 P d9a66fcfdb744bd6bd7db41019abd08f: No bootstrap required, opened a new log
I20260923 17:25:11.040647 20216 tablet_bootstrap.cc:492] T 00b2ad32c9c04893a426e5c44f84ccf8 P 494144bdd0314b94aa649bab67334db3: No bootstrap required, opened a new log
I20260923 17:25:11.040868 20215 ts_tablet_manager.cc:1397] T 00b2ad32c9c04893a426e5c44f84ccf8 P d9a66fcfdb744bd6bd7db41019abd08f: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260923 17:25:11.040874 20216 ts_tablet_manager.cc:1397] T 00b2ad32c9c04893a426e5c44f84ccf8 P 494144bdd0314b94aa649bab67334db3: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260923 17:25:11.042109 20217 tablet_bootstrap.cc:492] T 00b2ad32c9c04893a426e5c44f84ccf8 P 72e833a06f154110a6a8ac4625fac572: No bootstrap required, opened a new log
I20260923 17:25:11.042300 20217 ts_tablet_manager.cc:1397] T 00b2ad32c9c04893a426e5c44f84ccf8 P 72e833a06f154110a6a8ac4625fac572: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260923 17:25:11.044984 20215 raft_consensus.cc:348] T 00b2ad32c9c04893a426e5c44f84ccf8 P d9a66fcfdb744bd6bd7db41019abd08f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } } peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } }
I20260923 17:25:11.045246 20215 raft_consensus.cc:374] T 00b2ad32c9c04893a426e5c44f84ccf8 P d9a66fcfdb744bd6bd7db41019abd08f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:11.045323 20215 raft_consensus.cc:729] T 00b2ad32c9c04893a426e5c44f84ccf8 P d9a66fcfdb744bd6bd7db41019abd08f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9a66fcfdb744bd6bd7db41019abd08f, State: Initialized, Role: FOLLOWER
I20260923 17:25:11.045789 20215 consensus_queue.cc:260] T 00b2ad32c9c04893a426e5c44f84ccf8 P d9a66fcfdb744bd6bd7db41019abd08f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } } peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } }
I20260923 17:25:11.045714 20216 raft_consensus.cc:348] T 00b2ad32c9c04893a426e5c44f84ccf8 P 494144bdd0314b94aa649bab67334db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } } peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } }
I20260923 17:25:11.045990 20216 raft_consensus.cc:374] T 00b2ad32c9c04893a426e5c44f84ccf8 P 494144bdd0314b94aa649bab67334db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:11.046068 20216 raft_consensus.cc:729] T 00b2ad32c9c04893a426e5c44f84ccf8 P 494144bdd0314b94aa649bab67334db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 494144bdd0314b94aa649bab67334db3, State: Initialized, Role: FOLLOWER
I20260923 17:25:11.046536 20187 heartbeater.cc:503] Master 127.11.229.62:42063 was elected leader, sending a full tablet report...
I20260923 17:25:11.046511 20217 raft_consensus.cc:348] T 00b2ad32c9c04893a426e5c44f84ccf8 P 72e833a06f154110a6a8ac4625fac572 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } } peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } }
I20260923 17:25:11.046784 20217 raft_consensus.cc:374] T 00b2ad32c9c04893a426e5c44f84ccf8 P 72e833a06f154110a6a8ac4625fac572 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:11.046859 20217 raft_consensus.cc:729] T 00b2ad32c9c04893a426e5c44f84ccf8 P 72e833a06f154110a6a8ac4625fac572 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72e833a06f154110a6a8ac4625fac572, State: Initialized, Role: FOLLOWER
I20260923 17:25:11.046893 20215 ts_tablet_manager.cc:1428] T 00b2ad32c9c04893a426e5c44f84ccf8 P d9a66fcfdb744bd6bd7db41019abd08f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260923 17:25:11.047308 20217 consensus_queue.cc:260] T 00b2ad32c9c04893a426e5c44f84ccf8 P 72e833a06f154110a6a8ac4625fac572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } } peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } }
I20260923 17:25:11.047339 20216 consensus_queue.cc:260] T 00b2ad32c9c04893a426e5c44f84ccf8 P 494144bdd0314b94aa649bab67334db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } } peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } }
I20260923 17:25:11.048286 20216 ts_tablet_manager.cc:1428] T 00b2ad32c9c04893a426e5c44f84ccf8 P 494144bdd0314b94aa649bab67334db3: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260923 17:25:11.048307 20217 ts_tablet_manager.cc:1428] T 00b2ad32c9c04893a426e5c44f84ccf8 P 72e833a06f154110a6a8ac4625fac572: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 17:25:11.048676 19907 heartbeater.cc:503] Master 127.11.229.62:42063 was elected leader, sending a full tablet report...
I20260923 17:25:11.048676 20047 heartbeater.cc:503] Master 127.11.229.62:42063 was elected leader, sending a full tablet report...
I20260923 17:25:11.111181 20221 raft_consensus.cc:482] T 00b2ad32c9c04893a426e5c44f84ccf8 P d9a66fcfdb744bd6bd7db41019abd08f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:25:11.111495 20221 raft_consensus.cc:504] T 00b2ad32c9c04893a426e5c44f84ccf8 P d9a66fcfdb744bd6bd7db41019abd08f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } } peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } }
I20260923 17:25:11.113279 20221 leader_election.cc:288] T 00b2ad32c9c04893a426e5c44f84ccf8 P d9a66fcfdb744bd6bd7db41019abd08f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72e833a06f154110a6a8ac4625fac572 (127.11.229.2:33753), 494144bdd0314b94aa649bab67334db3 (127.11.229.1:34095)
I20260923 17:25:11.117671 20224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:11.113529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:34095 (local address 127.11.229.3:38223)
0923 17:25:11.114114 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:11.114129 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:11.114277 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:11.114562 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:11.114566 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:11.114583 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:11.114718 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:11.114725 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.115497 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.115499 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:11.116333 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:11.116342 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.117125 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.117128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:11.117280 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:11.117311 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 17:25:11.117401 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 17:25:11.117478 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":147,"spinlock_wait_cycles":38528,"thread_start_us":117,"threads_started":1}
I20260923 17:25:11.117698 20214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:11.113689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:38223 (local address 127.11.229.1:34095)
0923 17:25:11.113825 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:11.113830 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:11.114253 (+ 423us) server_negotiation.cc:407] Connection header received
0923 17:25:11.114391 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:11.114394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:11.114457 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:11.114537 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:11.114837 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.115379 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.116477 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.117003 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.117098 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:11.117141 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:11.117468 (+ 327us) server_negotiation.cc:299] Negotiation successful
0923 17:25:11.117552 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:25:11.117862 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:11.113529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:33753 (local address 127.11.229.3:41041)
0923 17:25:11.113705 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:11.113723 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:11.113831 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:11.114184 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:11.114187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:11.114267 (+ 80us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:11.114413 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:11.114421 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.115320 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.115324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:11.116681 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:11.116691 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.117464 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.117467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:11.117573 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:11.117594 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 17:25:11.117671 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 17:25:11.117747 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"spinlock_wait_cycles":4352}
I20260923 17:25:11.117959 20210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:11.113777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:41041 (local address 127.11.229.2:33753)
0923 17:25:11.113905 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:11.113909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:11.113927 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:25:11.113990 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:11.113993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:11.114044 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:11.114136 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:11.114562 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.115203 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.116806 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.117331 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.117445 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:11.117490 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:11.117712 (+ 222us) server_negotiation.cc:299] Negotiation successful
0923 17:25:11.117834 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 17:25:11.118629 20001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00b2ad32c9c04893a426e5c44f84ccf8" candidate_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72e833a06f154110a6a8ac4625fac572" is_pre_election: true
I20260923 17:25:11.118635 19861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00b2ad32c9c04893a426e5c44f84ccf8" candidate_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "494144bdd0314b94aa649bab67334db3" is_pre_election: true
I20260923 17:25:11.118957 20001 raft_consensus.cc:2454] T 00b2ad32c9c04893a426e5c44f84ccf8 P 72e833a06f154110a6a8ac4625fac572 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9a66fcfdb744bd6bd7db41019abd08f in term 0.
I20260923 17:25:11.118969 19861 raft_consensus.cc:2454] T 00b2ad32c9c04893a426e5c44f84ccf8 P 494144bdd0314b94aa649bab67334db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9a66fcfdb744bd6bd7db41019abd08f in term 0.
I20260923 17:25:11.119470 20072 leader_election.cc:302] T 00b2ad32c9c04893a426e5c44f84ccf8 P d9a66fcfdb744bd6bd7db41019abd08f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 494144bdd0314b94aa649bab67334db3, 72e833a06f154110a6a8ac4625fac572, d9a66fcfdb744bd6bd7db41019abd08f; no voters:
I20260923 17:25:11.119741 20221 raft_consensus.cc:2781] T 00b2ad32c9c04893a426e5c44f84ccf8 P d9a66fcfdb744bd6bd7db41019abd08f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:25:11.119839 20221 raft_consensus.cc:482] T 00b2ad32c9c04893a426e5c44f84ccf8 P d9a66fcfdb744bd6bd7db41019abd08f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:25:11.119920 20221 raft_consensus.cc:3037] T 00b2ad32c9c04893a426e5c44f84ccf8 P d9a66fcfdb744bd6bd7db41019abd08f [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:11.121047 20221 raft_consensus.cc:504] T 00b2ad32c9c04893a426e5c44f84ccf8 P d9a66fcfdb744bd6bd7db41019abd08f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } } peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } }
I20260923 17:25:11.121668 20221 leader_election.cc:288] T 00b2ad32c9c04893a426e5c44f84ccf8 P d9a66fcfdb744bd6bd7db41019abd08f [CANDIDATE]: Term 1 election: Requested vote from peers 72e833a06f154110a6a8ac4625fac572 (127.11.229.2:33753), 494144bdd0314b94aa649bab67334db3 (127.11.229.1:34095)
I20260923 17:25:11.121898 20001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00b2ad32c9c04893a426e5c44f84ccf8" candidate_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72e833a06f154110a6a8ac4625fac572"
I20260923 17:25:11.121914 19861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00b2ad32c9c04893a426e5c44f84ccf8" candidate_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "494144bdd0314b94aa649bab67334db3"
I20260923 17:25:11.122044 20001 raft_consensus.cc:3037] T 00b2ad32c9c04893a426e5c44f84ccf8 P 72e833a06f154110a6a8ac4625fac572 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:11.122073 19861 raft_consensus.cc:3037] T 00b2ad32c9c04893a426e5c44f84ccf8 P 494144bdd0314b94aa649bab67334db3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:11.123198 20001 raft_consensus.cc:2454] T 00b2ad32c9c04893a426e5c44f84ccf8 P 72e833a06f154110a6a8ac4625fac572 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9a66fcfdb744bd6bd7db41019abd08f in term 1.
I20260923 17:25:11.123347 19861 raft_consensus.cc:2454] T 00b2ad32c9c04893a426e5c44f84ccf8 P 494144bdd0314b94aa649bab67334db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9a66fcfdb744bd6bd7db41019abd08f in term 1.
I20260923 17:25:11.123591 20075 leader_election.cc:302] T 00b2ad32c9c04893a426e5c44f84ccf8 P d9a66fcfdb744bd6bd7db41019abd08f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72e833a06f154110a6a8ac4625fac572, d9a66fcfdb744bd6bd7db41019abd08f; no voters:
I20260923 17:25:11.123814 20221 raft_consensus.cc:2781] T 00b2ad32c9c04893a426e5c44f84ccf8 P d9a66fcfdb744bd6bd7db41019abd08f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:25:11.124083 20221 raft_consensus.cc:686] T 00b2ad32c9c04893a426e5c44f84ccf8 P d9a66fcfdb744bd6bd7db41019abd08f [term 1 LEADER]: Becoming Leader. State: Replica: d9a66fcfdb744bd6bd7db41019abd08f, State: Running, Role: LEADER
I20260923 17:25:11.124426 20221 consensus_queue.cc:237] T 00b2ad32c9c04893a426e5c44f84ccf8 P d9a66fcfdb744bd6bd7db41019abd08f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } } peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } }
I20260923 17:25:11.127907 19704 catalog_manager.cc:5495] T 00b2ad32c9c04893a426e5c44f84ccf8 P d9a66fcfdb744bd6bd7db41019abd08f reported cstate change: term changed from 0 to 1, leader changed from <none> to d9a66fcfdb744bd6bd7db41019abd08f (127.11.229.3). New cstate: current_term: 1 leader_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } health_report { overall_health: UNKNOWN } } }
I20260923 17:25:11.145015 20211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:11.141098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50540 (local address 127.11.229.3:41689)
0923 17:25:11.141249 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:11.141280 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:11.141353 (+ 73us) server_negotiation.cc:407] Connection header received
0923 17:25:11.141585 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:11.141588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:11.141643 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:11.141763 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:25:11.142038 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.142566 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.143460 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.143701 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.143809 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:11.144115 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:25:11.144587 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:25:11.144721 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:11.144785 (+ 64us) server_negotiation.cc:299] Negotiation successful
0923 17:25:11.144843 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 17:25:11.151234 19861 raft_consensus.cc:1261] T 00b2ad32c9c04893a426e5c44f84ccf8 P 494144bdd0314b94aa649bab67334db3 [term 1 FOLLOWER]: Refusing update from remote peer d9a66fcfdb744bd6bd7db41019abd08f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:25:11.151525 20001 raft_consensus.cc:1261] T 00b2ad32c9c04893a426e5c44f84ccf8 P 72e833a06f154110a6a8ac4625fac572 [term 1 FOLLOWER]: Refusing update from remote peer d9a66fcfdb744bd6bd7db41019abd08f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:25:11.152038 20221 consensus_queue.cc:1035] T 00b2ad32c9c04893a426e5c44f84ccf8 P d9a66fcfdb744bd6bd7db41019abd08f [LEADER]: Connected to new peer: Peer: permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:25:11.152242 20225 consensus_queue.cc:1035] T 00b2ad32c9c04893a426e5c44f84ccf8 P d9a66fcfdb744bd6bd7db41019abd08f [LEADER]: Connected to new peer: Peer: permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:25:11.160866 20230 mvcc.cc:205] Tried to move back new op lower bound from 7332594938467987456 to 7332594938368565248. Current Snapshot: MvccSnapshot[applied={T|T < 7332594938467987456}]
I20260923 17:25:11.161859 20229 mvcc.cc:205] Tried to move back new op lower bound from 7332594938467987456 to 7332594938368565248. Current Snapshot: MvccSnapshot[applied={T|T < 7332594938467987456}]
I20260923 17:25:11.170616 19703 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:25:11.170840 19703 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:25:11.173738 19703 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6dbd2dab05c43ae96c5126bcbfbf2b4: Sending DeleteTablet for 3 replicas of tablet 00b2ad32c9c04893a426e5c44f84ccf8
I20260923 17:25:11.174639 20121 tablet_service.cc:1514] Processing DeleteTablet for tablet 00b2ad32c9c04893a426e5c44f84ccf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:25:11 UTC) from {username='oryx', principal='oryx/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:50534
I20260923 17:25:11.174656 19981 tablet_service.cc:1514] Processing DeleteTablet for tablet 00b2ad32c9c04893a426e5c44f84ccf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:25:11 UTC) from {username='oryx', principal='oryx/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:57594
I20260923 17:25:11.175155 20238 tablet_replica.cc:330] T 00b2ad32c9c04893a426e5c44f84ccf8 P d9a66fcfdb744bd6bd7db41019abd08f: stopping tablet replica
I20260923 17:25:11.175190 20239 tablet_replica.cc:330] T 00b2ad32c9c04893a426e5c44f84ccf8 P 72e833a06f154110a6a8ac4625fac572: stopping tablet replica
I20260923 17:25:11.175370 19841 tablet_service.cc:1514] Processing DeleteTablet for tablet 00b2ad32c9c04893a426e5c44f84ccf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:25:11 UTC) from {username='oryx', principal='oryx/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:44786
I20260923 17:25:11.175570 20239 raft_consensus.cc:2229] T 00b2ad32c9c04893a426e5c44f84ccf8 P 72e833a06f154110a6a8ac4625fac572 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:25:11.175731 20238 raft_consensus.cc:2229] T 00b2ad32c9c04893a426e5c44f84ccf8 P d9a66fcfdb744bd6bd7db41019abd08f [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:25:11.175933 20239 raft_consensus.cc:2258] T 00b2ad32c9c04893a426e5c44f84ccf8 P 72e833a06f154110a6a8ac4625fac572 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:25:11.176472 20240 tablet_replica.cc:330] T 00b2ad32c9c04893a426e5c44f84ccf8 P 494144bdd0314b94aa649bab67334db3: stopping tablet replica
I20260923 17:25:11.176709 19703 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41448:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260923 17:25:11.176801 20238 raft_consensus.cc:2258] T 00b2ad32c9c04893a426e5c44f84ccf8 P d9a66fcfdb744bd6bd7db41019abd08f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:25:11.176769 20240 raft_consensus.cc:2229] T 00b2ad32c9c04893a426e5c44f84ccf8 P 494144bdd0314b94aa649bab67334db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:25:11.177145 20240 raft_consensus.cc:2258] T 00b2ad32c9c04893a426e5c44f84ccf8 P 494144bdd0314b94aa649bab67334db3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260923 17:25:11.177278 19703 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:25:11.178444 20239 ts_tablet_manager.cc:1905] T 00b2ad32c9c04893a426e5c44f84ccf8 P 72e833a06f154110a6a8ac4625fac572: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:25:11.178682 20238 ts_tablet_manager.cc:1905] T 00b2ad32c9c04893a426e5c44f84ccf8 P d9a66fcfdb744bd6bd7db41019abd08f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:25:11.179966 20240 ts_tablet_manager.cc:1905] T 00b2ad32c9c04893a426e5c44f84ccf8 P 494144bdd0314b94aa649bab67334db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:25:11.181612 20238 ts_tablet_manager.cc:1918] T 00b2ad32c9c04893a426e5c44f84ccf8 P d9a66fcfdb744bd6bd7db41019abd08f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:25:11.181715 20238 log.cc:1196] T 00b2ad32c9c04893a426e5c44f84ccf8 P d9a66fcfdb744bd6bd7db41019abd08f: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-2/wal/wals/00b2ad32c9c04893a426e5c44f84ccf8
I20260923 17:25:11.182062 20238 ts_tablet_manager.cc:1939] T 00b2ad32c9c04893a426e5c44f84ccf8 P d9a66fcfdb744bd6bd7db41019abd08f: Deleting consensus metadata
I20260923 17:25:11.182727 20240 ts_tablet_manager.cc:1918] T 00b2ad32c9c04893a426e5c44f84ccf8 P 494144bdd0314b94aa649bab67334db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:25:11.182822 20240 log.cc:1196] T 00b2ad32c9c04893a426e5c44f84ccf8 P 494144bdd0314b94aa649bab67334db3: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-0/wal/wals/00b2ad32c9c04893a426e5c44f84ccf8
I20260923 17:25:11.183121 20240 ts_tablet_manager.cc:1939] T 00b2ad32c9c04893a426e5c44f84ccf8 P 494144bdd0314b94aa649bab67334db3: Deleting consensus metadata
I20260923 17:25:11.183372 19688 catalog_manager.cc:4841] TS d9a66fcfdb744bd6bd7db41019abd08f (127.11.229.3:41689): tablet 00b2ad32c9c04893a426e5c44f84ccf8 (table test-table [id=2384f2864f814925a9814eae635d6afa]) successfully deleted
I20260923 17:25:11.184414 19688 catalog_manager.cc:4841] TS 494144bdd0314b94aa649bab67334db3 (127.11.229.1:34095): tablet 00b2ad32c9c04893a426e5c44f84ccf8 (table test-table [id=2384f2864f814925a9814eae635d6afa]) successfully deleted
I20260923 17:25:11.184463 19841 tablet_service.cc:1467] Processing CreateTablet for tablet bc821031fdf6402cb9d2ebba921f6aae (DEFAULT_TABLE table=test-table [id=cc6e75a308914200a98dbfa68d834fe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:25:11.184731 19841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc821031fdf6402cb9d2ebba921f6aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:11.184717 19981 tablet_service.cc:1467] Processing CreateTablet for tablet bc821031fdf6402cb9d2ebba921f6aae (DEFAULT_TABLE table=test-table [id=cc6e75a308914200a98dbfa68d834fe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:25:11.184898 20121 tablet_service.cc:1467] Processing CreateTablet for tablet bc821031fdf6402cb9d2ebba921f6aae (DEFAULT_TABLE table=test-table [id=cc6e75a308914200a98dbfa68d834fe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:25:11.185142 20121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc821031fdf6402cb9d2ebba921f6aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:11.185705 19981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc821031fdf6402cb9d2ebba921f6aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:11.187211 20216 tablet_bootstrap.cc:492] T bc821031fdf6402cb9d2ebba921f6aae P 494144bdd0314b94aa649bab67334db3: Bootstrap starting.
I20260923 17:25:11.187466 20215 tablet_bootstrap.cc:492] T bc821031fdf6402cb9d2ebba921f6aae P d9a66fcfdb744bd6bd7db41019abd08f: Bootstrap starting.
I20260923 17:25:11.188210 20216 tablet_bootstrap.cc:654] T bc821031fdf6402cb9d2ebba921f6aae P 494144bdd0314b94aa649bab67334db3: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:11.188470 20215 tablet_bootstrap.cc:654] T bc821031fdf6402cb9d2ebba921f6aae P d9a66fcfdb744bd6bd7db41019abd08f: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:11.188957 20217 tablet_bootstrap.cc:492] T bc821031fdf6402cb9d2ebba921f6aae P 72e833a06f154110a6a8ac4625fac572: Bootstrap starting.
I20260923 17:25:11.189965 20217 tablet_bootstrap.cc:654] T bc821031fdf6402cb9d2ebba921f6aae P 72e833a06f154110a6a8ac4625fac572: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:11.190433 20239 ts_tablet_manager.cc:1918] T 00b2ad32c9c04893a426e5c44f84ccf8 P 72e833a06f154110a6a8ac4625fac572: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:25:11.190590 20239 log.cc:1196] T 00b2ad32c9c04893a426e5c44f84ccf8 P 72e833a06f154110a6a8ac4625fac572: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-1/wal/wals/00b2ad32c9c04893a426e5c44f84ccf8
I20260923 17:25:11.190774 20215 tablet_bootstrap.cc:492] T bc821031fdf6402cb9d2ebba921f6aae P d9a66fcfdb744bd6bd7db41019abd08f: No bootstrap required, opened a new log
I20260923 17:25:11.190869 20215 ts_tablet_manager.cc:1397] T bc821031fdf6402cb9d2ebba921f6aae P d9a66fcfdb744bd6bd7db41019abd08f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 17:25:11.191092 20239 ts_tablet_manager.cc:1939] T 00b2ad32c9c04893a426e5c44f84ccf8 P 72e833a06f154110a6a8ac4625fac572: Deleting consensus metadata
I20260923 17:25:11.191515 20217 tablet_bootstrap.cc:492] T bc821031fdf6402cb9d2ebba921f6aae P 72e833a06f154110a6a8ac4625fac572: No bootstrap required, opened a new log
I20260923 17:25:11.191437 20215 raft_consensus.cc:348] T bc821031fdf6402cb9d2ebba921f6aae P d9a66fcfdb744bd6bd7db41019abd08f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } }
I20260923 17:25:11.191576 20215 raft_consensus.cc:374] T bc821031fdf6402cb9d2ebba921f6aae P d9a66fcfdb744bd6bd7db41019abd08f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:11.191605 20217 ts_tablet_manager.cc:1397] T bc821031fdf6402cb9d2ebba921f6aae P 72e833a06f154110a6a8ac4625fac572: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 17:25:11.191619 20215 raft_consensus.cc:729] T bc821031fdf6402cb9d2ebba921f6aae P d9a66fcfdb744bd6bd7db41019abd08f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9a66fcfdb744bd6bd7db41019abd08f, State: Initialized, Role: FOLLOWER
I20260923 17:25:11.191773 20215 consensus_queue.cc:260] T bc821031fdf6402cb9d2ebba921f6aae P d9a66fcfdb744bd6bd7db41019abd08f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } }
I20260923 17:25:11.192054 20215 ts_tablet_manager.cc:1428] T bc821031fdf6402cb9d2ebba921f6aae P d9a66fcfdb744bd6bd7db41019abd08f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 17:25:11.192045 20217 raft_consensus.cc:348] T bc821031fdf6402cb9d2ebba921f6aae P 72e833a06f154110a6a8ac4625fac572 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } }
I20260923 17:25:11.192179 20217 raft_consensus.cc:374] T bc821031fdf6402cb9d2ebba921f6aae P 72e833a06f154110a6a8ac4625fac572 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:11.192231 20217 raft_consensus.cc:729] T bc821031fdf6402cb9d2ebba921f6aae P 72e833a06f154110a6a8ac4625fac572 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72e833a06f154110a6a8ac4625fac572, State: Initialized, Role: FOLLOWER
I20260923 17:25:11.192366 20217 consensus_queue.cc:260] T bc821031fdf6402cb9d2ebba921f6aae P 72e833a06f154110a6a8ac4625fac572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } }
I20260923 17:25:11.192646 20217 ts_tablet_manager.cc:1428] T bc821031fdf6402cb9d2ebba921f6aae P 72e833a06f154110a6a8ac4625fac572: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 17:25:11.193092 19691 catalog_manager.cc:4841] TS 72e833a06f154110a6a8ac4625fac572 (127.11.229.2:33753): tablet 00b2ad32c9c04893a426e5c44f84ccf8 (table test-table [id=2384f2864f814925a9814eae635d6afa]) successfully deleted
I20260923 17:25:11.195092 20216 tablet_bootstrap.cc:492] T bc821031fdf6402cb9d2ebba921f6aae P 494144bdd0314b94aa649bab67334db3: No bootstrap required, opened a new log
I20260923 17:25:11.195211 20216 ts_tablet_manager.cc:1397] T bc821031fdf6402cb9d2ebba921f6aae P 494144bdd0314b94aa649bab67334db3: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20260923 17:25:11.195786 20216 raft_consensus.cc:348] T bc821031fdf6402cb9d2ebba921f6aae P 494144bdd0314b94aa649bab67334db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } }
I20260923 17:25:11.195884 20216 raft_consensus.cc:374] T bc821031fdf6402cb9d2ebba921f6aae P 494144bdd0314b94aa649bab67334db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:11.195906 20216 raft_consensus.cc:729] T bc821031fdf6402cb9d2ebba921f6aae P 494144bdd0314b94aa649bab67334db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 494144bdd0314b94aa649bab67334db3, State: Initialized, Role: FOLLOWER
I20260923 17:25:11.196072 20216 consensus_queue.cc:260] T bc821031fdf6402cb9d2ebba921f6aae P 494144bdd0314b94aa649bab67334db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } }
I20260923 17:25:11.196326 20216 ts_tablet_manager.cc:1428] T bc821031fdf6402cb9d2ebba921f6aae P 494144bdd0314b94aa649bab67334db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
W20260923 17:25:11.202247 20188 tablet.cc:2369] T bc821031fdf6402cb9d2ebba921f6aae P d9a66fcfdb744bd6bd7db41019abd08f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:25:11.230872 20048 tablet.cc:2369] T bc821031fdf6402cb9d2ebba921f6aae P 72e833a06f154110a6a8ac4625fac572: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:25:11.258548 19908 tablet.cc:2369] T bc821031fdf6402cb9d2ebba921f6aae P 494144bdd0314b94aa649bab67334db3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:25:11.470444 20223 raft_consensus.cc:482] T bc821031fdf6402cb9d2ebba921f6aae P 72e833a06f154110a6a8ac4625fac572 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:25:11.470618 20223 raft_consensus.cc:504] T bc821031fdf6402cb9d2ebba921f6aae P 72e833a06f154110a6a8ac4625fac572 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } }
I20260923 17:25:11.471865 20223 leader_election.cc:288] T bc821031fdf6402cb9d2ebba921f6aae P 72e833a06f154110a6a8ac4625fac572 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 494144bdd0314b94aa649bab67334db3 (127.11.229.1:34095), d9a66fcfdb744bd6bd7db41019abd08f (127.11.229.3:41689)
I20260923 17:25:11.476743 20214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:11.472251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:52481 (local address 127.11.229.1:34095)
0923 17:25:11.472392 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:11.472396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:11.472496 (+ 100us) server_negotiation.cc:407] Connection header received
0923 17:25:11.472654 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:11.472658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:11.472733 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:11.472846 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:11.473158 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.473748 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.475388 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.476040 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.476166 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:11.476217 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:11.476536 (+ 319us) server_negotiation.cc:299] Negotiation successful
0923 17:25:11.476609 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 17:25:11.476743 20211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:11.472662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:41113 (local address 127.11.229.3:41689)
0923 17:25:11.472809 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:11.472818 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:11.472857 (+ 39us) server_negotiation.cc:407] Connection header received
0923 17:25:11.473028 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:11.473033 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:11.473082 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:11.473171 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:11.473465 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.473996 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.475411 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.476033 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.476159 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:11.476217 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:11.476536 (+ 319us) server_negotiation.cc:299] Negotiation successful
0923 17:25:11.476606 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 17:25:11.476858 20244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:11.472120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:34095 (local address 127.11.229.2:52481)
0923 17:25:11.472379 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:11.472395 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:11.472532 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:11.472880 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:11.472884 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:11.472905 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:11.473037 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:11.473043 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.473860 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.473865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:11.475220 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:11.475234 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.476174 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.476178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:11.476363 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:11.476424 (+ 61us) client_negotiation.cc:769] Sending connection context
0923 17:25:11.476504 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 17:25:11.476631 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":127,"thread_start_us":98,"threads_started":1}
I20260923 17:25:11.477137 20245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:11.472519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:41689 (local address 127.11.229.2:41113)
0923 17:25:11.472787 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:11.472800 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:11.472899 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:11.473191 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:11.473193 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:11.473210 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:11.473341 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:11.473347 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.474128 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.474131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:11.475299 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:11.475310 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.476174 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.476178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:11.476367 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:11.476424 (+ 57us) client_negotiation.cc:769] Sending connection context
0923 17:25:11.476504 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 17:25:11.476660 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":80,"thread_start_us":58,"threads_started":1}
I20260923 17:25:11.477442 19861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc821031fdf6402cb9d2ebba921f6aae" candidate_uuid: "72e833a06f154110a6a8ac4625fac572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "494144bdd0314b94aa649bab67334db3" is_pre_election: true
I20260923 17:25:11.477619 19861 raft_consensus.cc:2454] T bc821031fdf6402cb9d2ebba921f6aae P 494144bdd0314b94aa649bab67334db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72e833a06f154110a6a8ac4625fac572 in term 0.
I20260923 17:25:11.477797 20141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc821031fdf6402cb9d2ebba921f6aae" candidate_uuid: "72e833a06f154110a6a8ac4625fac572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" is_pre_election: true
I20260923 17:25:11.477960 19932 leader_election.cc:302] T bc821031fdf6402cb9d2ebba921f6aae P 72e833a06f154110a6a8ac4625fac572 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 494144bdd0314b94aa649bab67334db3, 72e833a06f154110a6a8ac4625fac572; no voters:
I20260923 17:25:11.478055 20141 raft_consensus.cc:2454] T bc821031fdf6402cb9d2ebba921f6aae P d9a66fcfdb744bd6bd7db41019abd08f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72e833a06f154110a6a8ac4625fac572 in term 0.
I20260923 17:25:11.478209 20223 raft_consensus.cc:2781] T bc821031fdf6402cb9d2ebba921f6aae P 72e833a06f154110a6a8ac4625fac572 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:25:11.478281 20223 raft_consensus.cc:482] T bc821031fdf6402cb9d2ebba921f6aae P 72e833a06f154110a6a8ac4625fac572 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:25:11.478331 20223 raft_consensus.cc:3037] T bc821031fdf6402cb9d2ebba921f6aae P 72e833a06f154110a6a8ac4625fac572 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:11.479377 20223 raft_consensus.cc:504] T bc821031fdf6402cb9d2ebba921f6aae P 72e833a06f154110a6a8ac4625fac572 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } }
I20260923 17:25:11.479741 20223 leader_election.cc:288] T bc821031fdf6402cb9d2ebba921f6aae P 72e833a06f154110a6a8ac4625fac572 [CANDIDATE]: Term 1 election: Requested vote from peers 494144bdd0314b94aa649bab67334db3 (127.11.229.1:34095), d9a66fcfdb744bd6bd7db41019abd08f (127.11.229.3:41689)
I20260923 17:25:11.480131 19861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc821031fdf6402cb9d2ebba921f6aae" candidate_uuid: "72e833a06f154110a6a8ac4625fac572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "494144bdd0314b94aa649bab67334db3"
I20260923 17:25:11.480237 19861 raft_consensus.cc:3037] T bc821031fdf6402cb9d2ebba921f6aae P 494144bdd0314b94aa649bab67334db3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:11.481114 19861 raft_consensus.cc:2454] T bc821031fdf6402cb9d2ebba921f6aae P 494144bdd0314b94aa649bab67334db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72e833a06f154110a6a8ac4625fac572 in term 1.
I20260923 17:25:11.481423 19932 leader_election.cc:302] T bc821031fdf6402cb9d2ebba921f6aae P 72e833a06f154110a6a8ac4625fac572 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 494144bdd0314b94aa649bab67334db3, 72e833a06f154110a6a8ac4625fac572; no voters:
I20260923 17:25:11.481389 20141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc821031fdf6402cb9d2ebba921f6aae" candidate_uuid: "72e833a06f154110a6a8ac4625fac572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9a66fcfdb744bd6bd7db41019abd08f"
I20260923 17:25:11.481554 20141 raft_consensus.cc:3037] T bc821031fdf6402cb9d2ebba921f6aae P d9a66fcfdb744bd6bd7db41019abd08f [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:11.481622 20223 raft_consensus.cc:2781] T bc821031fdf6402cb9d2ebba921f6aae P 72e833a06f154110a6a8ac4625fac572 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:25:11.481824 20223 raft_consensus.cc:686] T bc821031fdf6402cb9d2ebba921f6aae P 72e833a06f154110a6a8ac4625fac572 [term 1 LEADER]: Becoming Leader. State: Replica: 72e833a06f154110a6a8ac4625fac572, State: Running, Role: LEADER
I20260923 17:25:11.482085 20223 consensus_queue.cc:237] T bc821031fdf6402cb9d2ebba921f6aae P 72e833a06f154110a6a8ac4625fac572 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } }
I20260923 17:25:11.482695 20141 raft_consensus.cc:2454] T bc821031fdf6402cb9d2ebba921f6aae P d9a66fcfdb744bd6bd7db41019abd08f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72e833a06f154110a6a8ac4625fac572 in term 1.
I20260923 17:25:11.484483 19702 catalog_manager.cc:5495] T bc821031fdf6402cb9d2ebba921f6aae P 72e833a06f154110a6a8ac4625fac572 reported cstate change: term changed from 0 to 1, leader changed from <none> to 72e833a06f154110a6a8ac4625fac572 (127.11.229.2). New cstate: current_term: 1 leader_uuid: "72e833a06f154110a6a8ac4625fac572" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } health_report { overall_health: UNKNOWN } } }
Sep 23 17:25:11 dist-test-slave-ptl9 krb5kdc[19659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184310, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.62@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM
I20260923 17:25:11.595561 20253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:11.587117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41462 (local address 127.11.229.62:42063)
0923 17:25:11.587401 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:11.587409 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:11.587606 (+ 197us) server_negotiation.cc:407] Connection header received
0923 17:25:11.587719 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:11.587722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:11.587764 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:11.587852 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:11.588554 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.589137 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.590006 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.590241 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.590327 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:11.592631 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:11.592635 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:11.592637 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:11.592666 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:11.594040 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:11.594476 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:11.594478 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:11.594479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:11.594555 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:11.594793 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:11.594795 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:11.594797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:11.594966 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:11.595041 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:11.595315 (+ 274us) server_negotiation.cc:299] Negotiation successful
0923 17:25:11.595411 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":114,"threads_started":1}
I20260923 17:25:11.595563 20254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:11.587079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:42063 (local address 127.0.0.1:41462)
0923 17:25:11.587519 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:11.587532 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:11.587634 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:11.587890 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:11.587892 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:11.588299 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:11.588403 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:11.588410 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.589240 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.589242 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:11.589823 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:11.589831 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.590349 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.590352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:11.590413 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:11.591003 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:11.591015 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:11.592499 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:11.594149 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:11.594158 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:11.594162 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:11.594390 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:11.594636 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:11.594638 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:11.594640 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:11.594716 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:11.595057 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:11.595062 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:11.595165 (+ 103us) client_negotiation.cc:769] Sending connection context
0923 17:25:11.595257 (+ 92us) client_negotiation.cc:240] Negotiation successful
0923 17:25:11.595366 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":140,"threads_started":1}
I20260923 17:25:11.599647 19702 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:41462:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 17:25:11.600432 19702 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:25:11.605855 19981 tablet_service.cc:1467] Processing CreateTablet for tablet fe2bfeb1bfbf4e2cac90dac9e271ad32 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=782fc1f79e904c10b895b0adf7770f05]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:25:11.605856 19841 tablet_service.cc:1467] Processing CreateTablet for tablet fe2bfeb1bfbf4e2cac90dac9e271ad32 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=782fc1f79e904c10b895b0adf7770f05]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:25:11.606000 20121 tablet_service.cc:1467] Processing CreateTablet for tablet fe2bfeb1bfbf4e2cac90dac9e271ad32 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=782fc1f79e904c10b895b0adf7770f05]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:25:11.606287 19841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe2bfeb1bfbf4e2cac90dac9e271ad32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:11.606284 19981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe2bfeb1bfbf4e2cac90dac9e271ad32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:11.606443 20121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe2bfeb1bfbf4e2cac90dac9e271ad32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:11.609020 20215 tablet_bootstrap.cc:492] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P d9a66fcfdb744bd6bd7db41019abd08f: Bootstrap starting.
I20260923 17:25:11.609232 20216 tablet_bootstrap.cc:492] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P 494144bdd0314b94aa649bab67334db3: Bootstrap starting.
I20260923 17:25:11.609505 20217 tablet_bootstrap.cc:492] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P 72e833a06f154110a6a8ac4625fac572: Bootstrap starting.
I20260923 17:25:11.609999 20215 tablet_bootstrap.cc:654] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P d9a66fcfdb744bd6bd7db41019abd08f: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:11.610090 20216 tablet_bootstrap.cc:654] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P 494144bdd0314b94aa649bab67334db3: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:11.610459 20217 tablet_bootstrap.cc:654] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P 72e833a06f154110a6a8ac4625fac572: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:11.611352 20216 tablet_bootstrap.cc:492] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P 494144bdd0314b94aa649bab67334db3: No bootstrap required, opened a new log
I20260923 17:25:11.611439 20216 ts_tablet_manager.cc:1397] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P 494144bdd0314b94aa649bab67334db3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 17:25:11.611606 20215 tablet_bootstrap.cc:492] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P d9a66fcfdb744bd6bd7db41019abd08f: No bootstrap required, opened a new log
I20260923 17:25:11.611685 20215 ts_tablet_manager.cc:1397] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P d9a66fcfdb744bd6bd7db41019abd08f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 17:25:11.611807 20217 tablet_bootstrap.cc:492] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P 72e833a06f154110a6a8ac4625fac572: No bootstrap required, opened a new log
I20260923 17:25:11.611912 20217 ts_tablet_manager.cc:1397] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P 72e833a06f154110a6a8ac4625fac572: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 17:25:11.611899 20216 raft_consensus.cc:348] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P 494144bdd0314b94aa649bab67334db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } }
I20260923 17:25:11.612025 20216 raft_consensus.cc:374] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P 494144bdd0314b94aa649bab67334db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:11.612082 20216 raft_consensus.cc:729] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P 494144bdd0314b94aa649bab67334db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 494144bdd0314b94aa649bab67334db3, State: Initialized, Role: FOLLOWER
I20260923 17:25:11.612094 20215 raft_consensus.cc:348] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P d9a66fcfdb744bd6bd7db41019abd08f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } }
I20260923 17:25:11.612224 20215 raft_consensus.cc:374] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P d9a66fcfdb744bd6bd7db41019abd08f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:11.612260 20215 raft_consensus.cc:729] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P d9a66fcfdb744bd6bd7db41019abd08f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9a66fcfdb744bd6bd7db41019abd08f, State: Initialized, Role: FOLLOWER
I20260923 17:25:11.612247 20216 consensus_queue.cc:260] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P 494144bdd0314b94aa649bab67334db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } }
I20260923 17:25:11.612380 20215 consensus_queue.cc:260] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P d9a66fcfdb744bd6bd7db41019abd08f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } }
I20260923 17:25:11.612411 20217 raft_consensus.cc:348] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P 72e833a06f154110a6a8ac4625fac572 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } }
I20260923 17:25:11.612532 20216 ts_tablet_manager.cc:1428] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P 494144bdd0314b94aa649bab67334db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 17:25:11.612560 20217 raft_consensus.cc:374] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P 72e833a06f154110a6a8ac4625fac572 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:11.612608 20217 raft_consensus.cc:729] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P 72e833a06f154110a6a8ac4625fac572 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72e833a06f154110a6a8ac4625fac572, State: Initialized, Role: FOLLOWER
I20260923 17:25:11.612655 20215 ts_tablet_manager.cc:1428] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P d9a66fcfdb744bd6bd7db41019abd08f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 17:25:11.612747 20217 consensus_queue.cc:260] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P 72e833a06f154110a6a8ac4625fac572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } }
I20260923 17:25:11.612752 20222 tablet_replica.cc:439] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P 494144bdd0314b94aa649bab67334db3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } } }
I20260923 17:25:11.612903 20222 tablet_replica.cc:442] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P 494144bdd0314b94aa649bab67334db3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:25:11.612947 20234 tablet_replica.cc:439] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P d9a66fcfdb744bd6bd7db41019abd08f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } } }
I20260923 17:25:11.613020 20217 ts_tablet_manager.cc:1428] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P 72e833a06f154110a6a8ac4625fac572: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 17:25:11.613075 20234 tablet_replica.cc:442] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P d9a66fcfdb744bd6bd7db41019abd08f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:25:11.613510 20223 tablet_replica.cc:439] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P 72e833a06f154110a6a8ac4625fac572: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } } }
I20260923 17:25:11.613655 20223 tablet_replica.cc:442] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P 72e833a06f154110a6a8ac4625fac572: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:25:11.691370 20234 raft_consensus.cc:482] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P d9a66fcfdb744bd6bd7db41019abd08f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:25:11.691510 20234 raft_consensus.cc:504] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P d9a66fcfdb744bd6bd7db41019abd08f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } }
I20260923 17:25:11.691919 20234 leader_election.cc:288] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P d9a66fcfdb744bd6bd7db41019abd08f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 494144bdd0314b94aa649bab67334db3 (127.11.229.1:34095), 72e833a06f154110a6a8ac4625fac572 (127.11.229.2:33753)
I20260923 17:25:11.692378 19861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2bfeb1bfbf4e2cac90dac9e271ad32" candidate_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "494144bdd0314b94aa649bab67334db3" is_pre_election: true
I20260923 17:25:11.692401 20001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2bfeb1bfbf4e2cac90dac9e271ad32" candidate_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72e833a06f154110a6a8ac4625fac572" is_pre_election: true
I20260923 17:25:11.692514 19861 raft_consensus.cc:2454] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P 494144bdd0314b94aa649bab67334db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9a66fcfdb744bd6bd7db41019abd08f in term 0.
I20260923 17:25:11.692622 20001 raft_consensus.cc:2454] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P 72e833a06f154110a6a8ac4625fac572 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9a66fcfdb744bd6bd7db41019abd08f in term 0.
I20260923 17:25:11.692934 20072 leader_election.cc:302] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P d9a66fcfdb744bd6bd7db41019abd08f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 494144bdd0314b94aa649bab67334db3, d9a66fcfdb744bd6bd7db41019abd08f; no voters:
I20260923 17:25:11.693138 20234 raft_consensus.cc:2781] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P d9a66fcfdb744bd6bd7db41019abd08f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:25:11.693189 20234 raft_consensus.cc:482] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P d9a66fcfdb744bd6bd7db41019abd08f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:25:11.693215 20234 raft_consensus.cc:3037] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P d9a66fcfdb744bd6bd7db41019abd08f [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:11.694371 20234 raft_consensus.cc:504] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P d9a66fcfdb744bd6bd7db41019abd08f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } }
I20260923 17:25:11.694828 20234 leader_election.cc:288] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P d9a66fcfdb744bd6bd7db41019abd08f [CANDIDATE]: Term 1 election: Requested vote from peers 494144bdd0314b94aa649bab67334db3 (127.11.229.1:34095), 72e833a06f154110a6a8ac4625fac572 (127.11.229.2:33753)
I20260923 17:25:11.695220 19861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2bfeb1bfbf4e2cac90dac9e271ad32" candidate_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "494144bdd0314b94aa649bab67334db3"
I20260923 17:25:11.695263 20001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2bfeb1bfbf4e2cac90dac9e271ad32" candidate_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72e833a06f154110a6a8ac4625fac572"
I20260923 17:25:11.695338 19861 raft_consensus.cc:3037] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P 494144bdd0314b94aa649bab67334db3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:11.695433 20001 raft_consensus.cc:3037] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P 72e833a06f154110a6a8ac4625fac572 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:11.696326 20001 raft_consensus.cc:2454] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P 72e833a06f154110a6a8ac4625fac572 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9a66fcfdb744bd6bd7db41019abd08f in term 1.
I20260923 17:25:11.696331 19861 raft_consensus.cc:2454] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P 494144bdd0314b94aa649bab67334db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9a66fcfdb744bd6bd7db41019abd08f in term 1.
I20260923 17:25:11.696650 20075 leader_election.cc:302] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P d9a66fcfdb744bd6bd7db41019abd08f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72e833a06f154110a6a8ac4625fac572, d9a66fcfdb744bd6bd7db41019abd08f; no voters:
I20260923 17:25:11.696835 20234 raft_consensus.cc:2781] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P d9a66fcfdb744bd6bd7db41019abd08f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:25:11.697070 20234 raft_consensus.cc:686] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P d9a66fcfdb744bd6bd7db41019abd08f [term 1 LEADER]: Becoming Leader. State: Replica: d9a66fcfdb744bd6bd7db41019abd08f, State: Running, Role: LEADER
I20260923 17:25:11.697237 20234 consensus_queue.cc:237] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P d9a66fcfdb744bd6bd7db41019abd08f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } }
I20260923 17:25:11.697962 20258 tablet_replica.cc:439] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P d9a66fcfdb744bd6bd7db41019abd08f: TxnStatusTablet state changed. Reason: New leader d9a66fcfdb744bd6bd7db41019abd08f. Latest consensus state: current_term: 1 leader_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } } }
I20260923 17:25:11.698076 20258 tablet_replica.cc:442] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P d9a66fcfdb744bd6bd7db41019abd08f: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:25:11.698355 20260 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 17:25:11.698565 20260 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 17:25:11.699244 19703 catalog_manager.cc:5495] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P d9a66fcfdb744bd6bd7db41019abd08f reported cstate change: term changed from 0 to 1, leader changed from <none> to d9a66fcfdb744bd6bd7db41019abd08f (127.11.229.3). New cstate: current_term: 1 leader_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } health_report { overall_health: HEALTHY } } }
I20260923 17:25:11.711231 20211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:11.707875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50544 (local address 127.11.229.3:41689)
0923 17:25:11.708067 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:11.708070 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:11.708107 (+ 37us) server_negotiation.cc:407] Connection header received
0923 17:25:11.708149 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:11.708151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:11.708195 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:11.708274 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:25:11.708572 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.709083 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.710115 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.710341 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.710452 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:11.710753 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:25:11.710837 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:25:11.710924 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:11.711030 (+ 106us) server_negotiation.cc:299] Negotiation successful
0923 17:25:11.711082 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 17:25:11.711231 20254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:11.707759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:41689 (local address 127.0.0.1:50544)
0923 17:25:11.707905 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:11.707920 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:11.708026 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:11.708318 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:11.708321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:11.708329 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:25:11.708447 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:11.708452 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.709203 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.709208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:11.709982 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:11.709992 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.710454 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.710466 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:11.710556 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:11.710612 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:25:11.710930 (+ 318us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:25:11.710949 (+ 19us) client_negotiation.cc:769] Sending connection context
0923 17:25:11.711011 (+ 62us) client_negotiation.cc:240] Negotiation successful
0923 17:25:11.711097 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260923 17:25:11.714238 19861 raft_consensus.cc:1261] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P 494144bdd0314b94aa649bab67334db3 [term 1 FOLLOWER]: Refusing update from remote peer d9a66fcfdb744bd6bd7db41019abd08f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:25:11.714231 20001 raft_consensus.cc:1261] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P 72e833a06f154110a6a8ac4625fac572 [term 1 FOLLOWER]: Refusing update from remote peer d9a66fcfdb744bd6bd7db41019abd08f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:25:11.714632 20258 consensus_queue.cc:1035] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P d9a66fcfdb744bd6bd7db41019abd08f [LEADER]: Connected to new peer: Peer: permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:25:11.714794 20234 consensus_queue.cc:1035] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P d9a66fcfdb744bd6bd7db41019abd08f [LEADER]: Connected to new peer: Peer: permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:25:11.716225 20223 tablet_replica.cc:439] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P 72e833a06f154110a6a8ac4625fac572: TxnStatusTablet state changed. Reason: New leader d9a66fcfdb744bd6bd7db41019abd08f. Latest consensus state: current_term: 1 leader_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } } }
I20260923 17:25:11.716336 20223 tablet_replica.cc:442] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P 72e833a06f154110a6a8ac4625fac572: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:25:11.717162 20222 tablet_replica.cc:439] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P 494144bdd0314b94aa649bab67334db3: TxnStatusTablet state changed. Reason: New leader d9a66fcfdb744bd6bd7db41019abd08f. Latest consensus state: current_term: 1 leader_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } } }
I20260923 17:25:11.717293 20222 tablet_replica.cc:442] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P 494144bdd0314b94aa649bab67334db3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:25:11.719085 20222 tablet_replica.cc:439] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P 494144bdd0314b94aa649bab67334db3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } } }
I20260923 17:25:11.719178 20222 tablet_replica.cc:442] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P 494144bdd0314b94aa649bab67334db3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:25:11.719225 20258 tablet_replica.cc:439] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P d9a66fcfdb744bd6bd7db41019abd08f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } } }
I20260923 17:25:11.719336 20258 tablet_replica.cc:442] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P d9a66fcfdb744bd6bd7db41019abd08f: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:25:11.719434 20258 tablet_replica.cc:439] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P d9a66fcfdb744bd6bd7db41019abd08f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } } }
I20260923 17:25:11.719493 20258 tablet_replica.cc:442] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P d9a66fcfdb744bd6bd7db41019abd08f: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:25:11.721303 20223 tablet_replica.cc:439] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P 72e833a06f154110a6a8ac4625fac572: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } } }
I20260923 17:25:11.721446 20223 tablet_replica.cc:442] T fe2bfeb1bfbf4e2cac90dac9e271ad32 P 72e833a06f154110a6a8ac4625fac572: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:25:11.734814 20273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:11.730114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57606 (local address 127.11.229.2:33753)
0923 17:25:11.730327 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:11.730332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:11.730345 (+ 13us) server_negotiation.cc:407] Connection header received
0923 17:25:11.730416 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:11.730419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:11.730478 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:11.730609 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:25:11.730862 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.731676 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.732906 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.733337 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.733434 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:11.733745 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:25:11.734264 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:25:11.734375 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:11.734482 (+ 107us) server_negotiation.cc:299] Negotiation successful
0923 17:25:11.734547 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":76,"threads_started":1}
I20260923 17:25:11.746891 20211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:11.742587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:47651 (local address 127.11.229.3:41689)
0923 17:25:11.742713 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:11.742717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:11.742730 (+ 13us) server_negotiation.cc:407] Connection header received
0923 17:25:11.742779 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:11.742782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:11.742832 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:11.742923 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:25:11.743587 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.744354 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.745585 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.745934 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.746033 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:11.746418 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:25:11.746510 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:25:11.746582 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:11.746704 (+ 122us) server_negotiation.cc:299] Negotiation successful
0923 17:25:11.746760 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 17:25:11.746907 20276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:11.742226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:41689 (local address 127.11.229.2:47651)
0923 17:25:11.742509 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:11.742524 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:11.742611 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:11.742945 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:11.742948 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:11.742955 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:25:11.743455 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:11.743470 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.744473 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.744477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:11.745455 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:11.745473 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.746117 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.746121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:11.746222 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:11.746273 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:25:11.746608 (+ 335us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:25:11.746614 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 17:25:11.746680 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 17:25:11.746761 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":85,"threads_started":1}
I20260923 17:25:11.754542 19861 raft_consensus.cc:1261] T bc821031fdf6402cb9d2ebba921f6aae P 494144bdd0314b94aa649bab67334db3 [term 1 FOLLOWER]: Refusing update from remote peer 72e833a06f154110a6a8ac4625fac572: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:25:11.754674 20141 raft_consensus.cc:1261] T bc821031fdf6402cb9d2ebba921f6aae P d9a66fcfdb744bd6bd7db41019abd08f [term 1 FOLLOWER]: Refusing update from remote peer 72e833a06f154110a6a8ac4625fac572: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:25:11.755162 20223 consensus_queue.cc:1035] T bc821031fdf6402cb9d2ebba921f6aae P 72e833a06f154110a6a8ac4625fac572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:25:11.755369 20246 consensus_queue.cc:1035] T bc821031fdf6402cb9d2ebba921f6aae P 72e833a06f154110a6a8ac4625fac572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:25:11.777663 20273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:11.773308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:51815 (local address 127.11.229.2:33753)
0923 17:25:11.773418 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:11.773423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:11.773669 (+ 246us) server_negotiation.cc:407] Connection header received
0923 17:25:11.773831 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:11.773834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:11.773886 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:11.773960 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:25:11.774315 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.775102 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.776440 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.776674 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.776789 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:11.777091 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:25:11.777178 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:25:11.777222 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:11.777408 (+ 186us) server_negotiation.cc:299] Negotiation successful
0923 17:25:11.777469 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260923 17:25:11.778043 20280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:11.773170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:33753 (local address 127.11.229.3:51815)
0923 17:25:11.773590 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:11.773602 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:11.773726 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:11.774075 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:11.774078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:11.774088 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:25:11.774194 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:11.774199 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.775239 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.775242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:11.776298 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:11.776308 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.776782 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.776786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:11.776874 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:11.776932 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:25:11.777306 (+ 374us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:25:11.777311 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 17:25:11.777808 (+ 497us) client_negotiation.cc:240] Negotiation successful
0923 17:25:11.777902 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":129,"threads_started":1}
I20260923 17:25:11.806739 19704 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:25:11.806948 19704 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:25:11.809698 19704 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6dbd2dab05c43ae96c5126bcbfbf2b4: Sending DeleteTablet for 3 replicas of tablet bc821031fdf6402cb9d2ebba921f6aae
I20260923 17:25:11.810705 19841 tablet_service.cc:1514] Processing DeleteTablet for tablet bc821031fdf6402cb9d2ebba921f6aae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:25:11 UTC) from {username='oryx', principal='oryx/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:44786
I20260923 17:25:11.810936 19981 tablet_service.cc:1514] Processing DeleteTablet for tablet bc821031fdf6402cb9d2ebba921f6aae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:25:11 UTC) from {username='oryx', principal='oryx/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:57594
I20260923 17:25:11.811203 20121 tablet_service.cc:1514] Processing DeleteTablet for tablet bc821031fdf6402cb9d2ebba921f6aae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:25:11 UTC) from {username='oryx', principal='oryx/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:50534
I20260923 17:25:11.811307 20286 tablet_replica.cc:330] T bc821031fdf6402cb9d2ebba921f6aae P 72e833a06f154110a6a8ac4625fac572: stopping tablet replica
I20260923 17:25:11.811436 20286 raft_consensus.cc:2229] T bc821031fdf6402cb9d2ebba921f6aae P 72e833a06f154110a6a8ac4625fac572 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:25:11.811770 20286 raft_consensus.cc:2258] T bc821031fdf6402cb9d2ebba921f6aae P 72e833a06f154110a6a8ac4625fac572 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:25:11.812503 20286 ts_tablet_manager.cc:1905] T bc821031fdf6402cb9d2ebba921f6aae P 72e833a06f154110a6a8ac4625fac572: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:25:11.815755 20286 ts_tablet_manager.cc:1918] T bc821031fdf6402cb9d2ebba921f6aae P 72e833a06f154110a6a8ac4625fac572: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:25:11.815850 20286 log.cc:1196] T bc821031fdf6402cb9d2ebba921f6aae P 72e833a06f154110a6a8ac4625fac572: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-1/wal/wals/bc821031fdf6402cb9d2ebba921f6aae
I20260923 17:25:11.816139 20286 ts_tablet_manager.cc:1939] T bc821031fdf6402cb9d2ebba921f6aae P 72e833a06f154110a6a8ac4625fac572: Deleting consensus metadata
I20260923 17:25:11.816385 20285 tablet_replica.cc:330] T bc821031fdf6402cb9d2ebba921f6aae P 494144bdd0314b94aa649bab67334db3: stopping tablet replica
I20260923 17:25:11.816915 19691 catalog_manager.cc:4841] TS 72e833a06f154110a6a8ac4625fac572 (127.11.229.2:33753): tablet bc821031fdf6402cb9d2ebba921f6aae (table test-table [id=cc6e75a308914200a98dbfa68d834fe3]) successfully deleted
I20260923 17:25:11.817171 20285 raft_consensus.cc:2229] T bc821031fdf6402cb9d2ebba921f6aae P 494144bdd0314b94aa649bab67334db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:25:11.817338 20285 raft_consensus.cc:2258] T bc821031fdf6402cb9d2ebba921f6aae P 494144bdd0314b94aa649bab67334db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:25:11.819622 20287 tablet_replica.cc:330] T bc821031fdf6402cb9d2ebba921f6aae P d9a66fcfdb744bd6bd7db41019abd08f: stopping tablet replica
I20260923 17:25:11.819762 20287 raft_consensus.cc:2229] T bc821031fdf6402cb9d2ebba921f6aae P d9a66fcfdb744bd6bd7db41019abd08f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:25:11.819880 20287 raft_consensus.cc:2258] T bc821031fdf6402cb9d2ebba921f6aae P d9a66fcfdb744bd6bd7db41019abd08f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:25:11.820473 20285 ts_tablet_manager.cc:1905] T bc821031fdf6402cb9d2ebba921f6aae P 494144bdd0314b94aa649bab67334db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:25:11.823467 20285 ts_tablet_manager.cc:1918] T bc821031fdf6402cb9d2ebba921f6aae P 494144bdd0314b94aa649bab67334db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:25:11.823552 20285 log.cc:1196] T bc821031fdf6402cb9d2ebba921f6aae P 494144bdd0314b94aa649bab67334db3: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-0/wal/wals/bc821031fdf6402cb9d2ebba921f6aae
I20260923 17:25:11.823832 20285 ts_tablet_manager.cc:1939] T bc821031fdf6402cb9d2ebba921f6aae P 494144bdd0314b94aa649bab67334db3: Deleting consensus metadata
I20260923 17:25:11.824661 19688 catalog_manager.cc:4841] TS 494144bdd0314b94aa649bab67334db3 (127.11.229.1:34095): tablet bc821031fdf6402cb9d2ebba921f6aae (table test-table [id=cc6e75a308914200a98dbfa68d834fe3]) successfully deleted
I20260923 17:25:11.824954 20287 ts_tablet_manager.cc:1905] T bc821031fdf6402cb9d2ebba921f6aae P d9a66fcfdb744bd6bd7db41019abd08f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:25:11.827884 20287 ts_tablet_manager.cc:1918] T bc821031fdf6402cb9d2ebba921f6aae P d9a66fcfdb744bd6bd7db41019abd08f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:25:11.827965 20287 log.cc:1196] T bc821031fdf6402cb9d2ebba921f6aae P d9a66fcfdb744bd6bd7db41019abd08f: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-2/wal/wals/bc821031fdf6402cb9d2ebba921f6aae
I20260923 17:25:11.828219 20287 ts_tablet_manager.cc:1939] T bc821031fdf6402cb9d2ebba921f6aae P d9a66fcfdb744bd6bd7db41019abd08f: Deleting consensus metadata
I20260923 17:25:11.829185 19688 catalog_manager.cc:4841] TS d9a66fcfdb744bd6bd7db41019abd08f (127.11.229.3:41689): tablet bc821031fdf6402cb9d2ebba921f6aae (table test-table [id=cc6e75a308914200a98dbfa68d834fe3]) successfully deleted
Sep 23 17:25:11 dist-test-slave-ptl9 krb5kdc[19659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 17:25:11 dist-test-slave-ptl9 krb5kdc[19659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:11 dist-test-slave-ptl9 krb5kdc[19659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
W20260923 17:25:11.853979 20253 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:11.845935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41470 (local address 127.11.229.62:42063)
0923 17:25:11.846071 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:11.846075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:11.846358 (+ 283us) server_negotiation.cc:407] Connection header received
0923 17:25:11.846512 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:11.846516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:11.846576 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:11.846683 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:11.847582 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.848434 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.849706 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.850070 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.850174 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:11.853838 (+ 3664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41470: BlockingRecv error: recv got EOF from 127.0.0.1:41470 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 23 17:25:11 dist-test-slave-ptl9 krb5kdc[19659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM
I20260923 17:25:11.866740 20253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:11.855850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41474 (local address 127.11.229.62:42063)
0923 17:25:11.855969 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:11.855973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:11.856407 (+ 434us) server_negotiation.cc:407] Connection header received
0923 17:25:11.856512 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:11.856516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:11.856566 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:11.856663 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:11.857450 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.858256 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.859453 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:11.859812 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:11.859967 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:11.862907 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:11.862911 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:11.862914 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:11.862944 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:11.864777 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:11.865348 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:11.865352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:11.865354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:11.865447 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:11.865773 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:11.865776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:11.865778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:11.866027 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:11.866160 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:11.866428 (+ 268us) server_negotiation.cc:299] Negotiation successful
0923 17:25:11.866566 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 17:25:11.869933 19704 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:25:11.870424 19704 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:25:11.876025 20121 tablet_service.cc:1467] Processing CreateTablet for tablet 14b081127ffd4054a81820c42910774e (DEFAULT_TABLE table=test-table [id=262e59469666432e956b2ab07b873483]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:25:11.876348 20121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14b081127ffd4054a81820c42910774e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:11.876286 19981 tablet_service.cc:1467] Processing CreateTablet for tablet 14b081127ffd4054a81820c42910774e (DEFAULT_TABLE table=test-table [id=262e59469666432e956b2ab07b873483]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:25:11.876538 19981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14b081127ffd4054a81820c42910774e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:11.877838 19841 tablet_service.cc:1467] Processing CreateTablet for tablet 14b081127ffd4054a81820c42910774e (DEFAULT_TABLE table=test-table [id=262e59469666432e956b2ab07b873483]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:25:11.878126 19841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14b081127ffd4054a81820c42910774e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:11.878855 20215 tablet_bootstrap.cc:492] T 14b081127ffd4054a81820c42910774e P d9a66fcfdb744bd6bd7db41019abd08f: Bootstrap starting.
I20260923 17:25:11.878917 20217 tablet_bootstrap.cc:492] T 14b081127ffd4054a81820c42910774e P 72e833a06f154110a6a8ac4625fac572: Bootstrap starting.
I20260923 17:25:11.879726 20215 tablet_bootstrap.cc:654] T 14b081127ffd4054a81820c42910774e P d9a66fcfdb744bd6bd7db41019abd08f: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:11.879930 20217 tablet_bootstrap.cc:654] T 14b081127ffd4054a81820c42910774e P 72e833a06f154110a6a8ac4625fac572: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:11.880582 20216 tablet_bootstrap.cc:492] T 14b081127ffd4054a81820c42910774e P 494144bdd0314b94aa649bab67334db3: Bootstrap starting.
I20260923 17:25:11.881495 20216 tablet_bootstrap.cc:654] T 14b081127ffd4054a81820c42910774e P 494144bdd0314b94aa649bab67334db3: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:11.881595 20215 tablet_bootstrap.cc:492] T 14b081127ffd4054a81820c42910774e P d9a66fcfdb744bd6bd7db41019abd08f: No bootstrap required, opened a new log
I20260923 17:25:11.881670 20215 ts_tablet_manager.cc:1397] T 14b081127ffd4054a81820c42910774e P d9a66fcfdb744bd6bd7db41019abd08f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260923 17:25:11.882074 20215 raft_consensus.cc:348] T 14b081127ffd4054a81820c42910774e P d9a66fcfdb744bd6bd7db41019abd08f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } }
I20260923 17:25:11.882196 20215 raft_consensus.cc:374] T 14b081127ffd4054a81820c42910774e P d9a66fcfdb744bd6bd7db41019abd08f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:11.882233 20215 raft_consensus.cc:729] T 14b081127ffd4054a81820c42910774e P d9a66fcfdb744bd6bd7db41019abd08f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9a66fcfdb744bd6bd7db41019abd08f, State: Initialized, Role: FOLLOWER
I20260923 17:25:11.882351 20215 consensus_queue.cc:260] T 14b081127ffd4054a81820c42910774e P d9a66fcfdb744bd6bd7db41019abd08f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } }
I20260923 17:25:11.882578 20217 tablet_bootstrap.cc:492] T 14b081127ffd4054a81820c42910774e P 72e833a06f154110a6a8ac4625fac572: No bootstrap required, opened a new log
I20260923 17:25:11.882603 20215 ts_tablet_manager.cc:1428] T 14b081127ffd4054a81820c42910774e P d9a66fcfdb744bd6bd7db41019abd08f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260923 17:25:11.882678 20217 ts_tablet_manager.cc:1397] T 14b081127ffd4054a81820c42910774e P 72e833a06f154110a6a8ac4625fac572: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260923 17:25:11.883157 20216 tablet_bootstrap.cc:492] T 14b081127ffd4054a81820c42910774e P 494144bdd0314b94aa649bab67334db3: No bootstrap required, opened a new log
I20260923 17:25:11.883168 20217 raft_consensus.cc:348] T 14b081127ffd4054a81820c42910774e P 72e833a06f154110a6a8ac4625fac572 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } }
I20260923 17:25:11.883244 20216 ts_tablet_manager.cc:1397] T 14b081127ffd4054a81820c42910774e P 494144bdd0314b94aa649bab67334db3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 17:25:11.883276 20217 raft_consensus.cc:374] T 14b081127ffd4054a81820c42910774e P 72e833a06f154110a6a8ac4625fac572 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:11.883308 20217 raft_consensus.cc:729] T 14b081127ffd4054a81820c42910774e P 72e833a06f154110a6a8ac4625fac572 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72e833a06f154110a6a8ac4625fac572, State: Initialized, Role: FOLLOWER
I20260923 17:25:11.883437 20217 consensus_queue.cc:260] T 14b081127ffd4054a81820c42910774e P 72e833a06f154110a6a8ac4625fac572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } }
I20260923 17:25:11.883673 20216 raft_consensus.cc:348] T 14b081127ffd4054a81820c42910774e P 494144bdd0314b94aa649bab67334db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } }
I20260923 17:25:11.883739 20217 ts_tablet_manager.cc:1428] T 14b081127ffd4054a81820c42910774e P 72e833a06f154110a6a8ac4625fac572: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 17:25:11.883777 20216 raft_consensus.cc:374] T 14b081127ffd4054a81820c42910774e P 494144bdd0314b94aa649bab67334db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:11.883812 20216 raft_consensus.cc:729] T 14b081127ffd4054a81820c42910774e P 494144bdd0314b94aa649bab67334db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 494144bdd0314b94aa649bab67334db3, State: Initialized, Role: FOLLOWER
I20260923 17:25:11.883929 20216 consensus_queue.cc:260] T 14b081127ffd4054a81820c42910774e P 494144bdd0314b94aa649bab67334db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } }
I20260923 17:25:11.884251 20216 ts_tablet_manager.cc:1428] T 14b081127ffd4054a81820c42910774e P 494144bdd0314b94aa649bab67334db3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 17:25:12.078651 20281 raft_consensus.cc:482] T 14b081127ffd4054a81820c42910774e P 72e833a06f154110a6a8ac4625fac572 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:25:12.078801 20281 raft_consensus.cc:504] T 14b081127ffd4054a81820c42910774e P 72e833a06f154110a6a8ac4625fac572 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } }
I20260923 17:25:12.079195 20281 leader_election.cc:288] T 14b081127ffd4054a81820c42910774e P 72e833a06f154110a6a8ac4625fac572 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 494144bdd0314b94aa649bab67334db3 (127.11.229.1:34095), d9a66fcfdb744bd6bd7db41019abd08f (127.11.229.3:41689)
I20260923 17:25:12.079761 20141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14b081127ffd4054a81820c42910774e" candidate_uuid: "72e833a06f154110a6a8ac4625fac572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" is_pre_election: true
I20260923 17:25:12.079764 19861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14b081127ffd4054a81820c42910774e" candidate_uuid: "72e833a06f154110a6a8ac4625fac572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "494144bdd0314b94aa649bab67334db3" is_pre_election: true
I20260923 17:25:12.079934 20141 raft_consensus.cc:2454] T 14b081127ffd4054a81820c42910774e P d9a66fcfdb744bd6bd7db41019abd08f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72e833a06f154110a6a8ac4625fac572 in term 0.
I20260923 17:25:12.079938 19861 raft_consensus.cc:2454] T 14b081127ffd4054a81820c42910774e P 494144bdd0314b94aa649bab67334db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72e833a06f154110a6a8ac4625fac572 in term 0.
I20260923 17:25:12.080286 19932 leader_election.cc:302] T 14b081127ffd4054a81820c42910774e P 72e833a06f154110a6a8ac4625fac572 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 494144bdd0314b94aa649bab67334db3, 72e833a06f154110a6a8ac4625fac572; no voters:
I20260923 17:25:12.080494 20281 raft_consensus.cc:2781] T 14b081127ffd4054a81820c42910774e P 72e833a06f154110a6a8ac4625fac572 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:25:12.080562 20281 raft_consensus.cc:482] T 14b081127ffd4054a81820c42910774e P 72e833a06f154110a6a8ac4625fac572 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:25:12.080605 20281 raft_consensus.cc:3037] T 14b081127ffd4054a81820c42910774e P 72e833a06f154110a6a8ac4625fac572 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:12.081701 20281 raft_consensus.cc:504] T 14b081127ffd4054a81820c42910774e P 72e833a06f154110a6a8ac4625fac572 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } }
I20260923 17:25:12.082038 20281 leader_election.cc:288] T 14b081127ffd4054a81820c42910774e P 72e833a06f154110a6a8ac4625fac572 [CANDIDATE]: Term 1 election: Requested vote from peers 494144bdd0314b94aa649bab67334db3 (127.11.229.1:34095), d9a66fcfdb744bd6bd7db41019abd08f (127.11.229.3:41689)
I20260923 17:25:12.082396 19861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14b081127ffd4054a81820c42910774e" candidate_uuid: "72e833a06f154110a6a8ac4625fac572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "494144bdd0314b94aa649bab67334db3"
I20260923 17:25:12.082456 20141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14b081127ffd4054a81820c42910774e" candidate_uuid: "72e833a06f154110a6a8ac4625fac572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9a66fcfdb744bd6bd7db41019abd08f"
I20260923 17:25:12.082501 19861 raft_consensus.cc:3037] T 14b081127ffd4054a81820c42910774e P 494144bdd0314b94aa649bab67334db3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:12.082620 20141 raft_consensus.cc:3037] T 14b081127ffd4054a81820c42910774e P d9a66fcfdb744bd6bd7db41019abd08f [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:12.083503 19861 raft_consensus.cc:2454] T 14b081127ffd4054a81820c42910774e P 494144bdd0314b94aa649bab67334db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72e833a06f154110a6a8ac4625fac572 in term 1.
I20260923 17:25:12.083612 20141 raft_consensus.cc:2454] T 14b081127ffd4054a81820c42910774e P d9a66fcfdb744bd6bd7db41019abd08f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72e833a06f154110a6a8ac4625fac572 in term 1.
I20260923 17:25:12.083843 19932 leader_election.cc:302] T 14b081127ffd4054a81820c42910774e P 72e833a06f154110a6a8ac4625fac572 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 494144bdd0314b94aa649bab67334db3, 72e833a06f154110a6a8ac4625fac572; no voters:
I20260923 17:25:12.084021 20281 raft_consensus.cc:2781] T 14b081127ffd4054a81820c42910774e P 72e833a06f154110a6a8ac4625fac572 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:25:12.084120 20281 raft_consensus.cc:686] T 14b081127ffd4054a81820c42910774e P 72e833a06f154110a6a8ac4625fac572 [term 1 LEADER]: Becoming Leader. State: Replica: 72e833a06f154110a6a8ac4625fac572, State: Running, Role: LEADER
I20260923 17:25:12.084291 20281 consensus_queue.cc:237] T 14b081127ffd4054a81820c42910774e P 72e833a06f154110a6a8ac4625fac572 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } }
I20260923 17:25:12.086053 19703 catalog_manager.cc:5495] T 14b081127ffd4054a81820c42910774e P 72e833a06f154110a6a8ac4625fac572 reported cstate change: term changed from 0 to 1, leader changed from <none> to 72e833a06f154110a6a8ac4625fac572 (127.11.229.2). New cstate: current_term: 1 leader_uuid: "72e833a06f154110a6a8ac4625fac572" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } health_report { overall_health: UNKNOWN } } }
I20260923 17:25:12.124707 20273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:12.121360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57612 (local address 127.11.229.2:33753)
0923 17:25:12.121506 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:12.121510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:12.121524 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:25:12.121580 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:12.121582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:12.121631 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:12.121708 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:25:12.121918 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:12.122623 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:12.123594 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:12.123858 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:12.123953 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:12.124235 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:25:12.124313 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:25:12.124384 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:12.124532 (+ 148us) server_negotiation.cc:299] Negotiation successful
0923 17:25:12.124581 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 17:25:12.126916 20141 raft_consensus.cc:1261] T 14b081127ffd4054a81820c42910774e P d9a66fcfdb744bd6bd7db41019abd08f [term 1 FOLLOWER]: Refusing update from remote peer 72e833a06f154110a6a8ac4625fac572: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:25:12.127005 19861 raft_consensus.cc:1261] T 14b081127ffd4054a81820c42910774e P 494144bdd0314b94aa649bab67334db3 [term 1 FOLLOWER]: Refusing update from remote peer 72e833a06f154110a6a8ac4625fac572: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:25:12.127394 20281 consensus_queue.cc:1035] T 14b081127ffd4054a81820c42910774e P 72e833a06f154110a6a8ac4625fac572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:25:12.127576 20223 consensus_queue.cc:1035] T 14b081127ffd4054a81820c42910774e P 72e833a06f154110a6a8ac4625fac572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:25:12.131557 20263 mvcc.cc:205] Tried to move back new op lower bound from 7332594942467379200 to 7332594942298587136. Current Snapshot: MvccSnapshot[applied={T|T < 7332594942467379200}]
I20260923 17:25:12.135978 19703 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:25:12.136123 19703 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:25:12.138048 19703 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6dbd2dab05c43ae96c5126bcbfbf2b4: Sending DeleteTablet for 3 replicas of tablet 14b081127ffd4054a81820c42910774e
I20260923 17:25:12.138607 19981 tablet_service.cc:1514] Processing DeleteTablet for tablet 14b081127ffd4054a81820c42910774e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:25:12 UTC) from {username='oryx', principal='oryx/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:57594
I20260923 17:25:12.138652 19841 tablet_service.cc:1514] Processing DeleteTablet for tablet 14b081127ffd4054a81820c42910774e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:25:12 UTC) from {username='oryx', principal='oryx/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:44786
I20260923 17:25:12.138823 20286 tablet_replica.cc:330] T 14b081127ffd4054a81820c42910774e P 72e833a06f154110a6a8ac4625fac572: stopping tablet replica
I20260923 17:25:12.138860 20285 tablet_replica.cc:330] T 14b081127ffd4054a81820c42910774e P 494144bdd0314b94aa649bab67334db3: stopping tablet replica
I20260923 17:25:12.138957 20286 raft_consensus.cc:2229] T 14b081127ffd4054a81820c42910774e P 72e833a06f154110a6a8ac4625fac572 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:25:12.138957 20285 raft_consensus.cc:2229] T 14b081127ffd4054a81820c42910774e P 494144bdd0314b94aa649bab67334db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:25:12.139000 20121 tablet_service.cc:1514] Processing DeleteTablet for tablet 14b081127ffd4054a81820c42910774e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:25:12 UTC) from {username='oryx', principal='oryx/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:50534
I20260923 17:25:12.139074 20285 raft_consensus.cc:2258] T 14b081127ffd4054a81820c42910774e P 494144bdd0314b94aa649bab67334db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:25:12.139145 20286 raft_consensus.cc:2258] T 14b081127ffd4054a81820c42910774e P 72e833a06f154110a6a8ac4625fac572 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:25:12.139189 20287 tablet_replica.cc:330] T 14b081127ffd4054a81820c42910774e P d9a66fcfdb744bd6bd7db41019abd08f: stopping tablet replica
I20260923 17:25:12.139326 20287 raft_consensus.cc:2229] T 14b081127ffd4054a81820c42910774e P d9a66fcfdb744bd6bd7db41019abd08f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:25:12.139493 20287 raft_consensus.cc:2258] T 14b081127ffd4054a81820c42910774e P d9a66fcfdb744bd6bd7db41019abd08f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:25:12.139878 20285 ts_tablet_manager.cc:1905] T 14b081127ffd4054a81820c42910774e P 494144bdd0314b94aa649bab67334db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:25:12.139892 20286 ts_tablet_manager.cc:1905] T 14b081127ffd4054a81820c42910774e P 72e833a06f154110a6a8ac4625fac572: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:25:12.139883 19703 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:25:12.140306 19703 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:25:12.141542 20287 ts_tablet_manager.cc:1905] T 14b081127ffd4054a81820c42910774e P d9a66fcfdb744bd6bd7db41019abd08f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:25:12.142361 20285 ts_tablet_manager.cc:1918] T 14b081127ffd4054a81820c42910774e P 494144bdd0314b94aa649bab67334db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:25:12.142419 20285 log.cc:1196] T 14b081127ffd4054a81820c42910774e P 494144bdd0314b94aa649bab67334db3: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-0/wal/wals/14b081127ffd4054a81820c42910774e
I20260923 17:25:12.142521 20286 ts_tablet_manager.cc:1918] T 14b081127ffd4054a81820c42910774e P 72e833a06f154110a6a8ac4625fac572: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:25:12.142588 20286 log.cc:1196] T 14b081127ffd4054a81820c42910774e P 72e833a06f154110a6a8ac4625fac572: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-1/wal/wals/14b081127ffd4054a81820c42910774e
I20260923 17:25:12.142606 20285 ts_tablet_manager.cc:1939] T 14b081127ffd4054a81820c42910774e P 494144bdd0314b94aa649bab67334db3: Deleting consensus metadata
I20260923 17:25:12.142843 20286 ts_tablet_manager.cc:1939] T 14b081127ffd4054a81820c42910774e P 72e833a06f154110a6a8ac4625fac572: Deleting consensus metadata
I20260923 17:25:12.143224 19688 catalog_manager.cc:4841] TS 494144bdd0314b94aa649bab67334db3 (127.11.229.1:34095): tablet 14b081127ffd4054a81820c42910774e (table test-table [id=262e59469666432e956b2ab07b873483]) successfully deleted
I20260923 17:25:12.143523 19691 catalog_manager.cc:4841] TS 72e833a06f154110a6a8ac4625fac572 (127.11.229.2:33753): tablet 14b081127ffd4054a81820c42910774e (table test-table [id=262e59469666432e956b2ab07b873483]) successfully deleted
I20260923 17:25:12.144248 20287 ts_tablet_manager.cc:1918] T 14b081127ffd4054a81820c42910774e P d9a66fcfdb744bd6bd7db41019abd08f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:25:12.144321 20287 log.cc:1196] T 14b081127ffd4054a81820c42910774e P d9a66fcfdb744bd6bd7db41019abd08f: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-2/wal/wals/14b081127ffd4054a81820c42910774e
I20260923 17:25:12.144573 20287 ts_tablet_manager.cc:1939] T 14b081127ffd4054a81820c42910774e P d9a66fcfdb744bd6bd7db41019abd08f: Deleting consensus metadata
I20260923 17:25:12.145332 19688 catalog_manager.cc:4841] TS d9a66fcfdb744bd6bd7db41019abd08f (127.11.229.3:41689): tablet 14b081127ffd4054a81820c42910774e (table test-table [id=262e59469666432e956b2ab07b873483]) successfully deleted
I20260923 17:25:12.146941 20121 tablet_service.cc:1467] Processing CreateTablet for tablet b350ab1281394449a184114ece389f52 (DEFAULT_TABLE table=test-table [id=f5817f3aca8d4610b1432cdb051155a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:25:12.146938 19841 tablet_service.cc:1467] Processing CreateTablet for tablet b350ab1281394449a184114ece389f52 (DEFAULT_TABLE table=test-table [id=f5817f3aca8d4610b1432cdb051155a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:25:12.147109 19981 tablet_service.cc:1467] Processing CreateTablet for tablet b350ab1281394449a184114ece389f52 (DEFAULT_TABLE table=test-table [id=f5817f3aca8d4610b1432cdb051155a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:25:12.147198 20121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b350ab1281394449a184114ece389f52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:12.147198 19841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b350ab1281394449a184114ece389f52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:12.147372 19981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b350ab1281394449a184114ece389f52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:12.149119 20217 tablet_bootstrap.cc:492] T b350ab1281394449a184114ece389f52 P 72e833a06f154110a6a8ac4625fac572: Bootstrap starting.
I20260923 17:25:12.149405 20216 tablet_bootstrap.cc:492] T b350ab1281394449a184114ece389f52 P 494144bdd0314b94aa649bab67334db3: Bootstrap starting.
I20260923 17:25:12.149478 20215 tablet_bootstrap.cc:492] T b350ab1281394449a184114ece389f52 P d9a66fcfdb744bd6bd7db41019abd08f: Bootstrap starting.
I20260923 17:25:12.150045 20217 tablet_bootstrap.cc:654] T b350ab1281394449a184114ece389f52 P 72e833a06f154110a6a8ac4625fac572: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:12.150095 20215 tablet_bootstrap.cc:654] T b350ab1281394449a184114ece389f52 P d9a66fcfdb744bd6bd7db41019abd08f: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:12.150557 20216 tablet_bootstrap.cc:654] T b350ab1281394449a184114ece389f52 P 494144bdd0314b94aa649bab67334db3: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:12.151250 20215 tablet_bootstrap.cc:492] T b350ab1281394449a184114ece389f52 P d9a66fcfdb744bd6bd7db41019abd08f: No bootstrap required, opened a new log
I20260923 17:25:12.151347 20215 ts_tablet_manager.cc:1397] T b350ab1281394449a184114ece389f52 P d9a66fcfdb744bd6bd7db41019abd08f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260923 17:25:12.151468 20217 tablet_bootstrap.cc:492] T b350ab1281394449a184114ece389f52 P 72e833a06f154110a6a8ac4625fac572: No bootstrap required, opened a new log
I20260923 17:25:12.151551 20217 ts_tablet_manager.cc:1397] T b350ab1281394449a184114ece389f52 P 72e833a06f154110a6a8ac4625fac572: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 17:25:12.151784 20215 raft_consensus.cc:348] T b350ab1281394449a184114ece389f52 P d9a66fcfdb744bd6bd7db41019abd08f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } }
I20260923 17:25:12.151908 20215 raft_consensus.cc:374] T b350ab1281394449a184114ece389f52 P d9a66fcfdb744bd6bd7db41019abd08f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:12.151952 20215 raft_consensus.cc:729] T b350ab1281394449a184114ece389f52 P d9a66fcfdb744bd6bd7db41019abd08f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9a66fcfdb744bd6bd7db41019abd08f, State: Initialized, Role: FOLLOWER
I20260923 17:25:12.151911 20217 raft_consensus.cc:348] T b350ab1281394449a184114ece389f52 P 72e833a06f154110a6a8ac4625fac572 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } }
I20260923 17:25:12.152053 20217 raft_consensus.cc:374] T b350ab1281394449a184114ece389f52 P 72e833a06f154110a6a8ac4625fac572 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:12.152104 20217 raft_consensus.cc:729] T b350ab1281394449a184114ece389f52 P 72e833a06f154110a6a8ac4625fac572 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72e833a06f154110a6a8ac4625fac572, State: Initialized, Role: FOLLOWER
I20260923 17:25:12.152086 20215 consensus_queue.cc:260] T b350ab1281394449a184114ece389f52 P d9a66fcfdb744bd6bd7db41019abd08f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } }
I20260923 17:25:12.152235 20217 consensus_queue.cc:260] T b350ab1281394449a184114ece389f52 P 72e833a06f154110a6a8ac4625fac572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } }
I20260923 17:25:12.152390 20215 ts_tablet_manager.cc:1428] T b350ab1281394449a184114ece389f52 P d9a66fcfdb744bd6bd7db41019abd08f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260923 17:25:12.152513 20217 ts_tablet_manager.cc:1428] T b350ab1281394449a184114ece389f52 P 72e833a06f154110a6a8ac4625fac572: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 17:25:12.154064 20216 tablet_bootstrap.cc:492] T b350ab1281394449a184114ece389f52 P 494144bdd0314b94aa649bab67334db3: No bootstrap required, opened a new log
I20260923 17:25:12.154175 20216 ts_tablet_manager.cc:1397] T b350ab1281394449a184114ece389f52 P 494144bdd0314b94aa649bab67334db3: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260923 17:25:12.154641 20216 raft_consensus.cc:348] T b350ab1281394449a184114ece389f52 P 494144bdd0314b94aa649bab67334db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } }
I20260923 17:25:12.154779 20216 raft_consensus.cc:374] T b350ab1281394449a184114ece389f52 P 494144bdd0314b94aa649bab67334db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:12.154821 20216 raft_consensus.cc:729] T b350ab1281394449a184114ece389f52 P 494144bdd0314b94aa649bab67334db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 494144bdd0314b94aa649bab67334db3, State: Initialized, Role: FOLLOWER
I20260923 17:25:12.154965 20216 consensus_queue.cc:260] T b350ab1281394449a184114ece389f52 P 494144bdd0314b94aa649bab67334db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } }
I20260923 17:25:12.155275 20216 ts_tablet_manager.cc:1428] T b350ab1281394449a184114ece389f52 P 494144bdd0314b94aa649bab67334db3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 17:25:12.451107 20222 raft_consensus.cc:482] T b350ab1281394449a184114ece389f52 P 494144bdd0314b94aa649bab67334db3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:25:12.451272 20222 raft_consensus.cc:504] T b350ab1281394449a184114ece389f52 P 494144bdd0314b94aa649bab67334db3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } }
I20260923 17:25:12.452376 20222 leader_election.cc:288] T b350ab1281394449a184114ece389f52 P 494144bdd0314b94aa649bab67334db3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72e833a06f154110a6a8ac4625fac572 (127.11.229.2:33753), d9a66fcfdb744bd6bd7db41019abd08f (127.11.229.3:41689)
I20260923 17:25:12.457275 20273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:12.452708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:38959 (local address 127.11.229.2:33753)
0923 17:25:12.452843 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:12.452850 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:12.453026 (+ 176us) server_negotiation.cc:407] Connection header received
0923 17:25:12.453209 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:12.453211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:12.453326 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:12.453414 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:12.453772 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:12.454650 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:12.456108 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:12.456619 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:12.456733 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:12.456779 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:12.457029 (+ 250us) server_negotiation.cc:299] Negotiation successful
0923 17:25:12.457132 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 17:25:12.457268 20310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:12.452641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:33753 (local address 127.11.229.1:38959)
0923 17:25:12.452928 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:12.452942 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:12.453084 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:12.453449 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:12.453454 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:12.453474 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:12.453624 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:12.453629 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:12.454804 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:12.454807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:12.455980 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:12.455991 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:12.456775 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:12.456779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:12.456889 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:12.456916 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 17:25:12.456991 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 17:25:12.457071 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":42,"thread_start_us":81,"threads_started":1}
I20260923 17:25:12.457357 20312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:12.452643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:41689 (local address 127.11.229.1:36435)
0923 17:25:12.453269 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:12.453302 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:12.453413 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:12.453723 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:12.453725 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:12.453746 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:12.453864 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:12.453868 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:12.454809 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:12.454813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:12.455988 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:12.455999 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:12.456904 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:12.456907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:12.457012 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:12.457038 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 17:25:12.457113 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 17:25:12.457186 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":391,"thread_start_us":50,"threads_started":1}
I20260923 17:25:12.457398 20311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:12.452797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:36435 (local address 127.11.229.3:41689)
0923 17:25:12.453069 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:12.453073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:12.453362 (+ 289us) server_negotiation.cc:407] Connection header received
0923 17:25:12.453537 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:12.453541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:12.453596 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:12.453677 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:12.453967 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:12.454716 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:12.456141 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:12.456771 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:12.456891 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:12.456944 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:12.457147 (+ 203us) server_negotiation.cc:299] Negotiation successful
0923 17:25:12.457228 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":106,"threads_started":1}
I20260923 17:25:12.457979 20001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b350ab1281394449a184114ece389f52" candidate_uuid: "494144bdd0314b94aa649bab67334db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72e833a06f154110a6a8ac4625fac572" is_pre_election: true
I20260923 17:25:12.458139 20001 raft_consensus.cc:2454] T b350ab1281394449a184114ece389f52 P 72e833a06f154110a6a8ac4625fac572 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 494144bdd0314b94aa649bab67334db3 in term 0.
I20260923 17:25:12.458592 19795 leader_election.cc:302] T b350ab1281394449a184114ece389f52 P 494144bdd0314b94aa649bab67334db3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 494144bdd0314b94aa649bab67334db3, 72e833a06f154110a6a8ac4625fac572; no voters:
I20260923 17:25:12.458611 20141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b350ab1281394449a184114ece389f52" candidate_uuid: "494144bdd0314b94aa649bab67334db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" is_pre_election: true
I20260923 17:25:12.458758 20141 raft_consensus.cc:2454] T b350ab1281394449a184114ece389f52 P d9a66fcfdb744bd6bd7db41019abd08f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 494144bdd0314b94aa649bab67334db3 in term 0.
I20260923 17:25:12.458889 20222 raft_consensus.cc:2781] T b350ab1281394449a184114ece389f52 P 494144bdd0314b94aa649bab67334db3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:25:12.458954 20222 raft_consensus.cc:482] T b350ab1281394449a184114ece389f52 P 494144bdd0314b94aa649bab67334db3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:25:12.458992 20222 raft_consensus.cc:3037] T b350ab1281394449a184114ece389f52 P 494144bdd0314b94aa649bab67334db3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:12.460453 20222 raft_consensus.cc:504] T b350ab1281394449a184114ece389f52 P 494144bdd0314b94aa649bab67334db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } }
I20260923 17:25:12.460860 20222 leader_election.cc:288] T b350ab1281394449a184114ece389f52 P 494144bdd0314b94aa649bab67334db3 [CANDIDATE]: Term 1 election: Requested vote from peers 72e833a06f154110a6a8ac4625fac572 (127.11.229.2:33753), d9a66fcfdb744bd6bd7db41019abd08f (127.11.229.3:41689)
I20260923 17:25:12.461293 20141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b350ab1281394449a184114ece389f52" candidate_uuid: "494144bdd0314b94aa649bab67334db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9a66fcfdb744bd6bd7db41019abd08f"
I20260923 17:25:12.461440 20141 raft_consensus.cc:3037] T b350ab1281394449a184114ece389f52 P d9a66fcfdb744bd6bd7db41019abd08f [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:12.461414 20001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b350ab1281394449a184114ece389f52" candidate_uuid: "494144bdd0314b94aa649bab67334db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72e833a06f154110a6a8ac4625fac572"
I20260923 17:25:12.461544 20001 raft_consensus.cc:3037] T b350ab1281394449a184114ece389f52 P 72e833a06f154110a6a8ac4625fac572 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:12.462879 20141 raft_consensus.cc:2454] T b350ab1281394449a184114ece389f52 P d9a66fcfdb744bd6bd7db41019abd08f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 494144bdd0314b94aa649bab67334db3 in term 1.
I20260923 17:25:12.462984 20001 raft_consensus.cc:2454] T b350ab1281394449a184114ece389f52 P 72e833a06f154110a6a8ac4625fac572 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 494144bdd0314b94aa649bab67334db3 in term 1.
I20260923 17:25:12.463263 19792 leader_election.cc:302] T b350ab1281394449a184114ece389f52 P 494144bdd0314b94aa649bab67334db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 494144bdd0314b94aa649bab67334db3, d9a66fcfdb744bd6bd7db41019abd08f; no voters:
I20260923 17:25:12.463534 20222 raft_consensus.cc:2781] T b350ab1281394449a184114ece389f52 P 494144bdd0314b94aa649bab67334db3 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:25:12.463768 20222 raft_consensus.cc:686] T b350ab1281394449a184114ece389f52 P 494144bdd0314b94aa649bab67334db3 [term 1 LEADER]: Becoming Leader. State: Replica: 494144bdd0314b94aa649bab67334db3, State: Running, Role: LEADER
I20260923 17:25:12.464100 20222 consensus_queue.cc:237] T b350ab1281394449a184114ece389f52 P 494144bdd0314b94aa649bab67334db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } }
I20260923 17:25:12.467069 19704 catalog_manager.cc:5495] T b350ab1281394449a184114ece389f52 P 494144bdd0314b94aa649bab67334db3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 494144bdd0314b94aa649bab67334db3 (127.11.229.1). New cstate: current_term: 1 leader_uuid: "494144bdd0314b94aa649bab67334db3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494144bdd0314b94aa649bab67334db3" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 } health_report { overall_health: UNKNOWN } } }
I20260923 17:25:12.527375 20316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:12.523084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44800 (local address 127.11.229.1:34095)
0923 17:25:12.523409 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:12.523418 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:12.523432 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:25:12.523519 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:12.523522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:12.523586 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:12.523717 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:25:12.523922 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:12.524772 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:12.525726 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:12.526012 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:12.526108 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:12.526356 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:25:12.527023 (+ 667us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:25:12.527132 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:12.527205 (+ 73us) server_negotiation.cc:299] Negotiation successful
0923 17:25:12.527257 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":130,"threads_started":1}
I20260923 17:25:12.537883 20311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:12.534729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:40675 (local address 127.11.229.3:41689)
0923 17:25:12.534881 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:12.534884 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:12.535006 (+ 122us) server_negotiation.cc:407] Connection header received
0923 17:25:12.535116 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:12.535119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:12.535167 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:12.535241 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:25:12.535478 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:12.535980 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:12.536823 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:12.537047 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:12.537130 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:12.537490 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:25:12.537565 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:25:12.537622 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:12.537727 (+ 105us) server_negotiation.cc:299] Negotiation successful
0923 17:25:12.537775 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 17:25:12.537901 20319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:12.534614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:41689 (local address 127.11.229.1:40675)
0923 17:25:12.534916 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:12.534937 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:12.535027 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:12.535264 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:12.535266 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:12.535273 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:25:12.535377 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:12.535381 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:12.536074 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:12.536076 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:12.536705 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:12.536713 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:12.537173 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:12.537175 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:12.537244 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:12.537367 (+ 123us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:25:12.537645 (+ 278us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:25:12.537650 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 17:25:12.537701 (+ 51us) client_negotiation.cc:240] Negotiation successful
0923 17:25:12.537767 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":118,"threads_started":1}
I20260923 17:25:12.544530 20001 raft_consensus.cc:1261] T b350ab1281394449a184114ece389f52 P 72e833a06f154110a6a8ac4625fac572 [term 1 FOLLOWER]: Refusing update from remote peer 494144bdd0314b94aa649bab67334db3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:25:12.544531 20141 raft_consensus.cc:1261] T b350ab1281394449a184114ece389f52 P d9a66fcfdb744bd6bd7db41019abd08f [term 1 FOLLOWER]: Refusing update from remote peer 494144bdd0314b94aa649bab67334db3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:25:12.545037 20222 consensus_queue.cc:1035] T b350ab1281394449a184114ece389f52 P 494144bdd0314b94aa649bab67334db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9a66fcfdb744bd6bd7db41019abd08f" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:25:12.545235 20313 consensus_queue.cc:1035] T b350ab1281394449a184114ece389f52 P 494144bdd0314b94aa649bab67334db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72e833a06f154110a6a8ac4625fac572" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 33753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:25:12.564970 20316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:12.561435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:34441 (local address 127.11.229.1:34095)
0923 17:25:12.561552 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:12.561557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:12.561750 (+ 193us) server_negotiation.cc:407] Connection header received
0923 17:25:12.561859 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:12.561863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:12.561935 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:12.562016 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:25:12.562255 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:12.562755 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:12.563653 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:12.564028 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:12.564113 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:12.564450 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:25:12.564542 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:25:12.564613 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:12.564762 (+ 149us) server_negotiation.cc:299] Negotiation successful
0923 17:25:12.564822 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260923 17:25:12.565065 20325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:12.561406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:34095 (local address 127.11.229.3:34441)
0923 17:25:12.561664 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:12.561683 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:12.561780 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:12.562050 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:12.562054 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:12.562062 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:25:12.562174 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:12.562179 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:12.562848 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:12.562850 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:12.563513 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:12.563521 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:12.564150 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:12.564154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:12.564303 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:12.564340 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:25:12.564646 (+ 306us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:25:12.564652 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 17:25:12.564739 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 17:25:12.564835 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":106,"threads_started":1}
I20260923 17:25:12.589573 19703 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:25:12.589738 19703 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:25:12.592139 19703 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6dbd2dab05c43ae96c5126bcbfbf2b4: Sending DeleteTablet for 3 replicas of tablet b350ab1281394449a184114ece389f52
I20260923 17:25:12.593230 19981 tablet_service.cc:1514] Processing DeleteTablet for tablet b350ab1281394449a184114ece389f52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:25:12 UTC) from {username='oryx', principal='oryx/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:57594
I20260923 17:25:12.593382 19841 tablet_service.cc:1514] Processing DeleteTablet for tablet b350ab1281394449a184114ece389f52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:25:12 UTC) from {username='oryx', principal='oryx/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:44786
I20260923 17:25:12.593502 20286 tablet_replica.cc:330] T b350ab1281394449a184114ece389f52 P 72e833a06f154110a6a8ac4625fac572: stopping tablet replica
I20260923 17:25:12.593585 20285 tablet_replica.cc:330] T b350ab1281394449a184114ece389f52 P 494144bdd0314b94aa649bab67334db3: stopping tablet replica
I20260923 17:25:12.593628 20286 raft_consensus.cc:2229] T b350ab1281394449a184114ece389f52 P 72e833a06f154110a6a8ac4625fac572 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:25:12.593731 20286 raft_consensus.cc:2258] T b350ab1281394449a184114ece389f52 P 72e833a06f154110a6a8ac4625fac572 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:25:12.593711 20285 raft_consensus.cc:2229] T b350ab1281394449a184114ece389f52 P 494144bdd0314b94aa649bab67334db3 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:25:12.594039 20285 raft_consensus.cc:2258] T b350ab1281394449a184114ece389f52 P 494144bdd0314b94aa649bab67334db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:25:12.594676 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 19776
I20260923 17:25:12.594720 20286 ts_tablet_manager.cc:1905] T b350ab1281394449a184114ece389f52 P 72e833a06f154110a6a8ac4625fac572: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:25:12.594913 20285 ts_tablet_manager.cc:1905] T b350ab1281394449a184114ece389f52 P 494144bdd0314b94aa649bab67334db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:25:12.595856 20121 tablet_service.cc:1514] Processing DeleteTablet for tablet b350ab1281394449a184114ece389f52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:25:12 UTC) from {username='oryx', principal='oryx/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:50534
I20260923 17:25:12.596056 20287 tablet_replica.cc:330] T b350ab1281394449a184114ece389f52 P d9a66fcfdb744bd6bd7db41019abd08f: stopping tablet replica
I20260923 17:25:12.596212 20287 raft_consensus.cc:2229] T b350ab1281394449a184114ece389f52 P d9a66fcfdb744bd6bd7db41019abd08f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:25:12.596385 20287 raft_consensus.cc:2258] T b350ab1281394449a184114ece389f52 P d9a66fcfdb744bd6bd7db41019abd08f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:25:12.597292 20287 ts_tablet_manager.cc:1905] T b350ab1281394449a184114ece389f52 P d9a66fcfdb744bd6bd7db41019abd08f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:25:12.598369 20286 ts_tablet_manager.cc:1918] T b350ab1281394449a184114ece389f52 P 72e833a06f154110a6a8ac4625fac572: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:25:12.598469 20286 log.cc:1196] T b350ab1281394449a184114ece389f52 P 72e833a06f154110a6a8ac4625fac572: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-1/wal/wals/b350ab1281394449a184114ece389f52
I20260923 17:25:12.598739 20286 ts_tablet_manager.cc:1939] T b350ab1281394449a184114ece389f52 P 72e833a06f154110a6a8ac4625fac572: Deleting consensus metadata
I20260923 17:25:12.599613 19691 catalog_manager.cc:4841] TS 72e833a06f154110a6a8ac4625fac572 (127.11.229.2:33753): tablet b350ab1281394449a184114ece389f52 (table test-table [id=f5817f3aca8d4610b1432cdb051155a3]) successfully deleted
I20260923 17:25:12.600400 20287 ts_tablet_manager.cc:1918] T b350ab1281394449a184114ece389f52 P d9a66fcfdb744bd6bd7db41019abd08f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:25:12.600497 20287 log.cc:1196] T b350ab1281394449a184114ece389f52 P d9a66fcfdb744bd6bd7db41019abd08f: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184271597484-12180-0/minicluster-data/ts-2/wal/wals/b350ab1281394449a184114ece389f52
I20260923 17:25:12.600752 20287 ts_tablet_manager.cc:1939] T b350ab1281394449a184114ece389f52 P d9a66fcfdb744bd6bd7db41019abd08f: Deleting consensus metadata
I20260923 17:25:12.601521 19688 catalog_manager.cc:4841] TS d9a66fcfdb744bd6bd7db41019abd08f (127.11.229.3:41689): tablet b350ab1281394449a184114ece389f52 (table test-table [id=f5817f3aca8d4610b1432cdb051155a3]) successfully deleted
W20260923 17:25:12.604033 19688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.229.1:34095 (error 108)
I20260923 17:25:12.604157 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 19916
I20260923 17:25:12.605834 20329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:12.605337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:34095 (local address 127.0.0.1:44806)
0923 17:25:12.605599 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:12.605695 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.1:34095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":60,"threads_started":1}
W20260923 17:25:12.606070 19688 catalog_manager.cc:4584] TS 494144bdd0314b94aa649bab67334db3 (127.11.229.1:34095): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b350ab1281394449a184114ece389f52: Network error: Client connection negotiation failed: client connection to 127.11.229.1:34095: connect: Connection refused (error 111)
I20260923 17:25:12.611917 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 20056
I20260923 17:25:12.619086 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 19672
2026-09-23T17:25:12Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3777 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:25:12 dist-test-slave-ptl9 krb5kdc[20332](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:25:12 dist-test-slave-ptl9 krb5kdc[20332](info): set up 2 sockets
Sep 23 17:25:12 dist-test-slave-ptl9 krb5kdc[20332](info): commencing operation
krb5kdc: starting...
W20260923 17:25:13.666313 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:25:13 dist-test-slave-ptl9 krb5kdc[20332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:25:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:25:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.229.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
I20260923 17:25:13.792943 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:46697
--webserver_interface=127.11.229.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:38941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:46697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:25:13.909910 20345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:13.910264 20345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:13.910348 20345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:13.914573 20345 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:25:13.914667 20345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:13.914706 20345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:13.914767 20345 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:25:13.914804 20345 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:25:13.920451 20345 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:38941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:46697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:46697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:25:13.921701 20345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:13.922773 20345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:13.931810 20351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:13.931948 20350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:13.932088 20345 server_base.cc:1034] running on GCE node
W20260923 17:25:13.931833 20353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:13.932878 20345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:13.934254 20345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:13.935432 20345 hybrid_clock.cc:648] HybridClock initialized: now 1790184313935416 us; error 33 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-23T17:25:13Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1320 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:25:14 dist-test-slave-ptl9 krb5kdc[20358](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:25:14 dist-test-slave-ptl9 krb5kdc[20358](info): set up 2 sockets
Sep 23 17:25:14 dist-test-slave-ptl9 krb5kdc[20358](info): commencing operation
krb5kdc: starting...
W20260923 17:25:14.990353 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:25:15 dist-test-slave-ptl9 krb5kdc[20358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:25:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:25:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.229.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
I20260923 17:25:15.114161 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:34307
--webserver_interface=127.11.229.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:44481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:34307
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:25:15.227274 20371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:15.227553 20371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:15.227596 20371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:15.231775 20371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:25:15.231833 20371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:15.231853 20371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:15.231869 20371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:25:15.231884 20371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:25:15.237102 20371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:44481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:34307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:34307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:25:15.238234 20371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:15.239254 20371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 17:25:15.248078 20371 server_base.cc:1034] running on GCE node
W20260923 17:25:15.248046 20379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:15.248041 20377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:15.248041 20376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:15.248941 20371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:15.250138 20371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:15.251319 20371 hybrid_clock.cc:648] HybridClock initialized: now 1790184315251296 us; error 45 us; skew 500 ppm
Sep 23 17:25:15 dist-test-slave-ptl9 krb5kdc[20358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184315, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:15.260453 20371 init.cc:377] Logged in from keytab as kudu/127.11.229.62@KRBTEST.COM (short username kudu)
I20260923 17:25:15.261518 20371 webserver.cc:460] Webserver started at http://127.11.229.62:34379/ using document root <none> and password file <none>
I20260923 17:25:15.262130 20371 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:15.262192 20371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:15.262414 20371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:25:15.264526 20371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "1828ed9af465494eb3c18c3570bb5558"
format_stamp: "Formatted at 2026-09-23 17:25:15 on dist-test-slave-ptl9"
server_key: "12a2f11e3e011b42d4172fd2969c1b96"
server_key_iv: "99abbf82c1e5104cbc827c290c0d713c"
server_key_version: "encryptionkey@0"
I20260923 17:25:15.265200 20371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "1828ed9af465494eb3c18c3570bb5558"
format_stamp: "Formatted at 2026-09-23 17:25:15 on dist-test-slave-ptl9"
server_key: "12a2f11e3e011b42d4172fd2969c1b96"
server_key_iv: "99abbf82c1e5104cbc827c290c0d713c"
server_key_version: "encryptionkey@0"
I20260923 17:25:15.269004 20371 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:25:15.271494 20386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:15.272797 20371 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260923 17:25:15.272938 20371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "1828ed9af465494eb3c18c3570bb5558"
format_stamp: "Formatted at 2026-09-23 17:25:15 on dist-test-slave-ptl9"
server_key: "12a2f11e3e011b42d4172fd2969c1b96"
server_key_iv: "99abbf82c1e5104cbc827c290c0d713c"
server_key_version: "encryptionkey@0"
I20260923 17:25:15.273049 20371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:15.287988 20371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:15.291025 20371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:15.291231 20371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:15.298858 20371 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:34307
I20260923 17:25:15.298877 20438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:34307 every 8 connection(s)
I20260923 17:25:15.299819 20371 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:25:15.300938 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 20371
I20260923 17:25:15.301090 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:25:15.301410 12180 external_mini_cluster.cc:1428] Setting key 3888db34142b3168fe3d05f8bcb631bc
I20260923 17:25:15.303153 20439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:15.310424 20439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558: Bootstrap starting.
Sep 23 17:25:15 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:25:15.313730 20439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:15.314679 20439 log.cc:824] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558: Log is configured to *not* fsync() on all Append() calls
I20260923 17:25:15.317176 20439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558: No bootstrap required, opened a new log
I20260923 17:25:15.317883 20442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:15.302841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55844 (local address 127.11.229.62:34307)
0923 17:25:15.303271 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:15.303290 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:15.303333 (+ 43us) server_negotiation.cc:407] Connection header received
0923 17:25:15.304356 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:15.304381 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:15.304821 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:15.305193 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:15.305966 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:15.307492 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:15.308343 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:15.309046 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:15.309222 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:15.312258 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:15.312265 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:15.312283 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:15.312318 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:15.314563 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:15.315051 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:15.315061 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:15.315072 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:15.315223 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:15.315568 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:15.315573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:15.315575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:15.316088 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:15.316284 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:15.316530 (+ 246us) server_negotiation.cc:299] Negotiation successful
0923 17:25:15.316835 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":116,"threads_started":1}
I20260923 17:25:15.321466 20439 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1828ed9af465494eb3c18c3570bb5558" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34307 } }
I20260923 17:25:15.321700 20439 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:15.321735 20439 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1828ed9af465494eb3c18c3570bb5558, State: Initialized, Role: FOLLOWER
I20260923 17:25:15.322157 20439 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1828ed9af465494eb3c18c3570bb5558" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34307 } }
I20260923 17:25:15.322278 20439 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:25:15.322357 20439 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:25:15.322491 20439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:15.323448 20439 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1828ed9af465494eb3c18c3570bb5558" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34307 } }
I20260923 17:25:15.323843 20439 leader_election.cc:302] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1828ed9af465494eb3c18c3570bb5558; no voters:
I20260923 17:25:15.324227 20439 leader_election.cc:288] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:25:15.324357 20444 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:25:15.324600 20444 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558 [term 1 LEADER]: Becoming Leader. State: Replica: 1828ed9af465494eb3c18c3570bb5558, State: Running, Role: LEADER
I20260923 17:25:15.324939 20444 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1828ed9af465494eb3c18c3570bb5558" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34307 } }
I20260923 17:25:15.325574 20439 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:25:15.326844 20446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1828ed9af465494eb3c18c3570bb5558. Latest consensus state: current_term: 1 leader_uuid: "1828ed9af465494eb3c18c3570bb5558" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1828ed9af465494eb3c18c3570bb5558" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34307 } } }
I20260923 17:25:15.326870 20445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1828ed9af465494eb3c18c3570bb5558" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1828ed9af465494eb3c18c3570bb5558" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34307 } } }
I20260923 17:25:15.326977 20446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558 [sys.catalog]: This master's current role is: LEADER
I20260923 17:25:15.326977 20445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558 [sys.catalog]: This master's current role is: LEADER
I20260923 17:25:15.327406 20453 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:25:15.330683 20453 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:25:15.336503 20453 catalog_manager.cc:1347] Generated new cluster ID: 962145a11a604efe83ce42b46d1dbbdf
I20260923 17:25:15.336581 20453 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:25:15.375203 20453 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:25:15.376237 20453 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:25:15.382294 20453 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558: Generated new TSK 0
I20260923 17:25:15.383031 20453 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.229.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab.
Entry for principal kudu/127.11.229.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab.
I20260923 17:25:15.420935 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.1:0
--local_ip_for_outbound_sockets=127.11.229.1
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:34307
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:44481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:25:15.537014 20465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:15.537325 20465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:15.537415 20465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:15.541435 20465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:15.541487 20465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:15.541584 20465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.1
I20260923 17:25:15.546907 20465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:44481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:34307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.1
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:25:15.548032 20465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:15.549091 20465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:15.558905 20471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:15.558900 20470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:15.558900 20473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:15.559093 20465 server_base.cc:1034] running on GCE node
I20260923 17:25:15.559597 20465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:15.560257 20465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:15.561408 20465 hybrid_clock.cc:648] HybridClock initialized: now 1790184315561373 us; error 54 us; skew 500 ppm
Sep 23 17:25:15 dist-test-slave-ptl9 krb5kdc[20358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184315, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:15.564146 20465 init.cc:377] Logged in from keytab as kudu/127.11.229.1@KRBTEST.COM (short username kudu)
I20260923 17:25:15.565008 20465 webserver.cc:460] Webserver started at http://127.11.229.1:34867/ using document root <none> and password file <none>
I20260923 17:25:15.565657 20465 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:15.565748 20465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:15.565989 20465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:25:15.568071 20465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-0/data/instance:
uuid: "14eab5e51552440eaad8fec14bcbe48a"
format_stamp: "Formatted at 2026-09-23 17:25:15 on dist-test-slave-ptl9"
server_key: "e08926fc578c4cd61aecce6ed61d242f"
server_key_iv: "c38c590b8fa42bd907a491ad58ecba1f"
server_key_version: "encryptionkey@0"
I20260923 17:25:15.568656 20465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance:
uuid: "14eab5e51552440eaad8fec14bcbe48a"
format_stamp: "Formatted at 2026-09-23 17:25:15 on dist-test-slave-ptl9"
server_key: "e08926fc578c4cd61aecce6ed61d242f"
server_key_iv: "c38c590b8fa42bd907a491ad58ecba1f"
server_key_version: "encryptionkey@0"
I20260923 17:25:15.572407 20465 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260923 17:25:15.574862 20480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:15.576164 20465 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:25:15.576290 20465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-0/wal
uuid: "14eab5e51552440eaad8fec14bcbe48a"
format_stamp: "Formatted at 2026-09-23 17:25:15 on dist-test-slave-ptl9"
server_key: "e08926fc578c4cd61aecce6ed61d242f"
server_key_iv: "c38c590b8fa42bd907a491ad58ecba1f"
server_key_version: "encryptionkey@0"
I20260923 17:25:15.576403 20465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:15.592103 20465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:15.595619 20465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:15.595839 20465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:15.596494 20465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:25:15.597394 20465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:25:15.597466 20465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:15.597543 20465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:25:15.597585 20465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:15.606995 20465 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.1:38951
I20260923 17:25:15.607038 20593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.1:38951 every 8 connection(s)
I20260923 17:25:15.607975 20465 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
Sep 23 17:25:15 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184315, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:25:15.617926 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 20465
I20260923 17:25:15.618064 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance
I20260923 17:25:15.618408 12180 external_mini_cluster.cc:1428] Setting key caa30cd67da666fc30c6e444fc370e05
I20260923 17:25:15.622176 20442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:15.610138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:49355 (local address 127.11.229.62:34307)
0923 17:25:15.610305 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:15.610309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:15.611388 (+ 1079us) server_negotiation.cc:407] Connection header received
0923 17:25:15.612288 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:15.612290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:15.612340 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:15.612415 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:15.613818 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:15.614364 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:15.615586 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:15.615827 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:15.615904 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:15.618567 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:15.618571 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:15.618573 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:15.618599 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:15.620038 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:15.620557 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:15.620561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:15.620563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:15.620655 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:15.621007 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:15.621010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:15.621012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:15.621202 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:15.621332 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:15.621872 (+ 540us) server_negotiation.cc:299] Negotiation successful
0923 17:25:15.622009 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 17:25:15.623054 20596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:15.610515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.1:49355)
0923 17:25:15.611233 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:15.611276 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:15.612096 (+ 820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:15.612601 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:15.612608 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:15.613052 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:15.613667 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:15.613683 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:15.614496 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:15.614499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:15.615470 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:15.615480 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:15.615995 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:15.615999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:15.616158 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:15.616739 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:15.616756 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:15.618381 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:15.620165 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:15.620172 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:15.620189 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:15.620450 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:15.620748 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:15.620753 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:15.620755 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:15.620904 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:15.621332 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:15.621339 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:15.621613 (+ 274us) client_negotiation.cc:769] Sending connection context
0923 17:25:15.621819 (+ 206us) client_negotiation.cc:240] Negotiation successful
0923 17:25:15.622182 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":125,"threads_started":1}
I20260923 17:25:15.624495 20594 heartbeater.cc:347] Connected to a master server at 127.11.229.62:34307
I20260923 17:25:15.624764 20594 heartbeater.cc:464] Registering TS with master...
I20260923 17:25:15.625403 20594 heartbeater.cc:511] Master 127.11.229.62:34307 requested a full tablet report, sending...
I20260923 17:25:15.627364 20403 ts_manager.cc:194] Registered new tserver with Master: 14eab5e51552440eaad8fec14bcbe48a (127.11.229.1:38951)
WARNING: no policy specified for kudu/127.11.229.2@KRBTEST.COM; defaulting to no policy
I20260923 17:25:15.628963 20403 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.1@KRBTEST.COM'} at 127.11.229.1:49355
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab.
Entry for principal kudu/127.11.229.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab.
I20260923 17:25:15.646095 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.2:0
--local_ip_for_outbound_sockets=127.11.229.2
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:34307
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:44481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:25:15.760710 20599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:15.760993 20599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:15.761036 20599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:15.765139 20599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:15.765200 20599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:15.765321 20599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.2
I20260923 17:25:15.770704 20599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:44481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:34307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.2
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:25:15.771950 20599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:15.773093 20599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:15.783092 20607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:15.783102 20605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:15.783181 20604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:15.783363 20599 server_base.cc:1034] running on GCE node
I20260923 17:25:15.783862 20599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:15.784469 20599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:15.785625 20599 hybrid_clock.cc:648] HybridClock initialized: now 1790184315785590 us; error 53 us; skew 500 ppm
Sep 23 17:25:15 dist-test-slave-ptl9 krb5kdc[20358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184315, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:15.788372 20599 init.cc:377] Logged in from keytab as kudu/127.11.229.2@KRBTEST.COM (short username kudu)
I20260923 17:25:15.789330 20599 webserver.cc:460] Webserver started at http://127.11.229.2:41845/ using document root <none> and password file <none>
I20260923 17:25:15.789889 20599 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:15.789948 20599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:15.790120 20599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:25:15.792212 20599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-1/data/instance:
uuid: "55920ce52d6241cbbb94986388a50e57"
format_stamp: "Formatted at 2026-09-23 17:25:15 on dist-test-slave-ptl9"
server_key: "144d6d2e12f12c22c817510674a87702"
server_key_iv: "8bad6cc726ddfa3ed7ba40376207b8a8"
server_key_version: "encryptionkey@0"
I20260923 17:25:15.792753 20599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance:
uuid: "55920ce52d6241cbbb94986388a50e57"
format_stamp: "Formatted at 2026-09-23 17:25:15 on dist-test-slave-ptl9"
server_key: "144d6d2e12f12c22c817510674a87702"
server_key_iv: "8bad6cc726ddfa3ed7ba40376207b8a8"
server_key_version: "encryptionkey@0"
I20260923 17:25:15.796792 20599 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 17:25:15.799439 20614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:15.800854 20599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260923 17:25:15.801013 20599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-1/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-1/wal
uuid: "55920ce52d6241cbbb94986388a50e57"
format_stamp: "Formatted at 2026-09-23 17:25:15 on dist-test-slave-ptl9"
server_key: "144d6d2e12f12c22c817510674a87702"
server_key_iv: "8bad6cc726ddfa3ed7ba40376207b8a8"
server_key_version: "encryptionkey@0"
I20260923 17:25:15.801141 20599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:15.822038 20599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:15.825047 20599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:15.825255 20599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:15.826090 20599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:25:15.827240 20599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:25:15.827313 20599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:15.827371 20599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:25:15.827395 20599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:15.836826 20599 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.2:40359
I20260923 17:25:15.836843 20727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.2:40359 every 8 connection(s)
I20260923 17:25:15.837868 20599 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
I20260923 17:25:15.843189 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 20599
I20260923 17:25:15.843307 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance
I20260923 17:25:15.843629 12180 external_mini_cluster.cc:1428] Setting key 3e67470438db0608e23d7b2c5e825d28
Sep 23 17:25:15 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184315, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:25:15.852427 20442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:15.839749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:38901 (local address 127.11.229.62:34307)
0923 17:25:15.839900 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:15.839904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:15.841087 (+ 1183us) server_negotiation.cc:407] Connection header received
0923 17:25:15.842109 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:15.842112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:15.842164 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:15.842241 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:15.843761 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:15.844288 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:15.845800 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:15.846017 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:15.846105 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:15.848799 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:15.848802 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:15.848804 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:15.848828 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:15.850366 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:15.850887 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:15.850890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:15.850892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:15.850975 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:15.851321 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:15.851325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:15.851327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:15.851512 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:15.851626 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:15.852137 (+ 511us) server_negotiation.cc:299] Negotiation successful
0923 17:25:15.852263 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
WARNING: no policy specified for kudu/127.11.229.3@KRBTEST.COM; defaulting to no policy
I20260923 17:25:15.853365 20730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:15.840165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.2:38901)
0923 17:25:15.840869 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:15.840914 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:15.841830 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:15.842436 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:15.842444 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:15.842811 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:15.843588 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:15.843609 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:15.844422 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:15.844425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:15.845661 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:15.845677 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:15.846182 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:15.846189 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:15.846439 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:15.847149 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:15.847167 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:15.848620 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:15.850488 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:15.850494 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:15.850512 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:15.850777 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:15.851072 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:15.851076 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:15.851078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:15.851222 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:15.851624 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:15.851631 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:15.851888 (+ 257us) client_negotiation.cc:769] Sending connection context
0923 17:25:15.852096 (+ 208us) client_negotiation.cc:240] Negotiation successful
0923 17:25:15.852435 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":117,"threads_started":1}
I20260923 17:25:15.854651 20728 heartbeater.cc:347] Connected to a master server at 127.11.229.62:34307
I20260923 17:25:15.854940 20728 heartbeater.cc:464] Registering TS with master...
I20260923 17:25:15.855468 20728 heartbeater.cc:511] Master 127.11.229.62:34307 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.3@KRBTEST.COM" created.
I20260923 17:25:15.856686 20403 ts_manager.cc:194] Registered new tserver with Master: 55920ce52d6241cbbb94986388a50e57 (127.11.229.2:40359)
I20260923 17:25:15.857360 20403 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.2@KRBTEST.COM'} at 127.11.229.2:38901
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab.
Entry for principal kudu/127.11.229.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab.
I20260923 17:25:15.872236 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.3:0
--local_ip_for_outbound_sockets=127.11.229.3
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:34307
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:44481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:25:15.986660 20733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:15.986914 20733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:15.986958 20733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:15.991067 20733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:15.991120 20733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:15.991178 20733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.3
I20260923 17:25:15.996246 20733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:44481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:34307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.3
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:25:15.997242 20733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:15.998391 20733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:16.008723 20741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:16.008630 20738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:16.008630 20739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:16.008844 20733 server_base.cc:1034] running on GCE node
I20260923 17:25:16.009450 20733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:16.010205 20733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:16.011399 20733 hybrid_clock.cc:648] HybridClock initialized: now 1790184316011381 us; error 41 us; skew 500 ppm
Sep 23 17:25:16 dist-test-slave-ptl9 krb5kdc[20358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184316, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:16.014503 20733 init.cc:377] Logged in from keytab as kudu/127.11.229.3@KRBTEST.COM (short username kudu)
I20260923 17:25:16.015535 20733 webserver.cc:460] Webserver started at http://127.11.229.3:45171/ using document root <none> and password file <none>
I20260923 17:25:16.016166 20733 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:16.016264 20733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:16.016510 20733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:25:16.018811 20733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-2/data/instance:
uuid: "44b4389d5063459f928d788333ae2a7c"
format_stamp: "Formatted at 2026-09-23 17:25:16 on dist-test-slave-ptl9"
server_key: "1f06d836331f33f9826577021882a1b9"
server_key_iv: "1ab666be0deceaeea8586302ababb8d0"
server_key_version: "encryptionkey@0"
I20260923 17:25:16.019419 20733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance:
uuid: "44b4389d5063459f928d788333ae2a7c"
format_stamp: "Formatted at 2026-09-23 17:25:16 on dist-test-slave-ptl9"
server_key: "1f06d836331f33f9826577021882a1b9"
server_key_iv: "1ab666be0deceaeea8586302ababb8d0"
server_key_version: "encryptionkey@0"
I20260923 17:25:16.023478 20733 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:25:16.026346 20748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:16.027768 20733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:25:16.027913 20733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-2/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-2/wal
uuid: "44b4389d5063459f928d788333ae2a7c"
format_stamp: "Formatted at 2026-09-23 17:25:16 on dist-test-slave-ptl9"
server_key: "1f06d836331f33f9826577021882a1b9"
server_key_iv: "1ab666be0deceaeea8586302ababb8d0"
server_key_version: "encryptionkey@0"
I20260923 17:25:16.028038 20733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:16.047127 20733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:16.050769 20733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:16.050994 20733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:16.051666 20733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:25:16.052567 20733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:25:16.052641 20733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:16.052713 20733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:25:16.052765 20733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:16.062098 20733 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.3:42203
I20260923 17:25:16.062124 20861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.3:42203 every 8 connection(s)
I20260923 17:25:16.063099 20733 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
I20260923 17:25:16.068944 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 20733
I20260923 17:25:16.069058 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance
I20260923 17:25:16.069465 12180 external_mini_cluster.cc:1428] Setting key 352cf21c193519d3a84f5d2832a88b93
Sep 23 17:25:16 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184316, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:25:16.077521 20442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:16.065182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:37325 (local address 127.11.229.62:34307)
0923 17:25:16.065338 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:16.065342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:16.066462 (+ 1120us) server_negotiation.cc:407] Connection header received
0923 17:25:16.067310 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:16.067313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:16.067362 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:16.067435 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:16.068585 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.069194 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.070843 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.071158 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.071266 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:16.073760 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:16.073763 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:16.073765 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:16.073791 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:16.075352 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:16.075871 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:16.075873 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:16.075874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:16.075949 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:16.076269 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:16.076274 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:16.076279 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:16.076479 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:16.076559 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:16.077202 (+ 643us) server_negotiation.cc:299] Negotiation successful
0923 17:25:16.077387 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 17:25:16.078418 20864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:16.065618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.3:37325)
0923 17:25:16.066299 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:16.066340 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:16.067127 (+ 787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:16.067570 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:16.067577 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:16.067929 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:16.068432 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:16.068457 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.069382 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.069386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:16.070699 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:16.070710 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.071304 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.071307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:16.071450 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:16.072066 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:16.072082 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:16.073596 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:16.075491 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:16.075497 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:16.075510 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:16.075780 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:16.076028 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:16.076030 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:16.076032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:16.076162 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:16.076607 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:16.076613 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:16.076922 (+ 309us) client_negotiation.cc:769] Sending connection context
0923 17:25:16.077158 (+ 236us) client_negotiation.cc:240] Negotiation successful
0923 17:25:16.077532 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":121,"threads_started":1}
I20260923 17:25:16.079555 20862 heartbeater.cc:347] Connected to a master server at 127.11.229.62:34307
I20260923 17:25:16.079802 20862 heartbeater.cc:464] Registering TS with master...
I20260923 17:25:16.080401 20862 heartbeater.cc:511] Master 127.11.229.62:34307 requested a full tablet report, sending...
I20260923 17:25:16.081499 20403 ts_manager.cc:194] Registered new tserver with Master: 44b4389d5063459f928d788333ae2a7c (127.11.229.3:42203)
I20260923 17:25:16.082032 20403 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.3@KRBTEST.COM'} at 127.11.229.3:37325
I20260923 17:25:16.083752 12180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.11.229.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.229.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.229.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.1.keytab.
Entry for principal oryx/127.11.229.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.1.keytab.
WARNING: no policy specified for oryx/127.11.229.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.229.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.229.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.2.keytab.
Entry for principal oryx/127.11.229.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.2.keytab.
WARNING: no policy specified for oryx/127.11.229.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.229.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.229.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.3.keytab.
Entry for principal oryx/127.11.229.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.3.keytab.
I20260923 17:25:16.156309 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 20465
I20260923 17:25:16.162587 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 20599
I20260923 17:25:16.168447 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 20733
I20260923 17:25:16.173947 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 20371
I20260923 17:25:16.180454 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:34307
--webserver_interface=127.11.229.62
--webserver_port=34379
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:44481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:34307
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:25:16.292958 20871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:16.293247 20871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:16.293372 20871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:16.297338 20871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:25:16.297387 20871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:16.297407 20871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:16.297425 20871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:25:16.297441 20871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:25:16.302740 20871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:44481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:34307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:34307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=34379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:25:16.303799 20871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:16.304798 20871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:16.313547 20879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:16.313556 20877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:16.313829 20871 server_base.cc:1034] running on GCE node
W20260923 17:25:16.313558 20876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:16.314605 20871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:16.315788 20871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:16.316994 20871 hybrid_clock.cc:648] HybridClock initialized: now 1790184316316980 us; error 47 us; skew 500 ppm
Sep 23 17:25:16 dist-test-slave-ptl9 krb5kdc[20358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184316, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:16.319875 20871 init.cc:377] Logged in from keytab as kudu/127.11.229.62@KRBTEST.COM (short username kudu)
I20260923 17:25:16.320794 20871 webserver.cc:460] Webserver started at http://127.11.229.62:34379/ using document root <none> and password file <none>
I20260923 17:25:16.321466 20871 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:16.321591 20871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:16.325574 20871 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260923 17:25:16.327759 20886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:16.329062 20871 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:25:16.329216 20871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "1828ed9af465494eb3c18c3570bb5558"
format_stamp: "Formatted at 2026-09-23 17:25:15 on dist-test-slave-ptl9"
server_key: "12a2f11e3e011b42d4172fd2969c1b96"
server_key_iv: "99abbf82c1e5104cbc827c290c0d713c"
server_key_version: "encryptionkey@0"
I20260923 17:25:16.329844 20871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:16.339766 20871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:16.342756 20871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:16.342967 20871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:16.350395 20871 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:34307
I20260923 17:25:16.350425 20938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:34307 every 8 connection(s)
I20260923 17:25:16.351430 20871 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:25:16.355762 20939 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:25:16.356823 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 20871
I20260923 17:25:16.357366 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.1:38951
--local_ip_for_outbound_sockets=127.11.229.1
--tserver_master_addrs=127.11.229.62:34307
--webserver_port=34867
--webserver_interface=127.11.229.1
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:44481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.229.1
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:25:16.357854 20939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558: Bootstrap starting.
I20260923 17:25:16.375772 20939 log.cc:824] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558: Log is configured to *not* fsync() on all Append() calls
I20260923 17:25:16.382624 20939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:25:16.383006 20939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558: Bootstrap complete.
I20260923 17:25:16.387354 20939 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1828ed9af465494eb3c18c3570bb5558" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34307 } }
I20260923 17:25:16.387760 20939 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1828ed9af465494eb3c18c3570bb5558, State: Initialized, Role: FOLLOWER
I20260923 17:25:16.388273 20939 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1828ed9af465494eb3c18c3570bb5558" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34307 } }
I20260923 17:25:16.388386 20939 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:25:16.388471 20939 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:25:16.388569 20939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558 [term 1 FOLLOWER]: Advancing to term 2
I20260923 17:25:16.390595 20939 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1828ed9af465494eb3c18c3570bb5558" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34307 } }
I20260923 17:25:16.390933 20939 leader_election.cc:302] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1828ed9af465494eb3c18c3570bb5558; no voters:
I20260923 17:25:16.391296 20939 leader_election.cc:288] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 17:25:16.391384 20943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 17:25:16.391685 20943 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558 [term 2 LEADER]: Becoming Leader. State: Replica: 1828ed9af465494eb3c18c3570bb5558, State: Running, Role: LEADER
I20260923 17:25:16.391981 20943 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1828ed9af465494eb3c18c3570bb5558" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34307 } }
I20260923 17:25:16.392549 20939 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:25:16.392956 20943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1828ed9af465494eb3c18c3570bb5558. Latest consensus state: current_term: 2 leader_uuid: "1828ed9af465494eb3c18c3570bb5558" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1828ed9af465494eb3c18c3570bb5558" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34307 } } }
I20260923 17:25:16.393067 20943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558 [sys.catalog]: This master's current role is: LEADER
I20260923 17:25:16.393455 20944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1828ed9af465494eb3c18c3570bb5558" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1828ed9af465494eb3c18c3570bb5558" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34307 } } }
I20260923 17:25:16.393646 20944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558 [sys.catalog]: This master's current role is: LEADER
I20260923 17:25:16.393468 20947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:25:16.397701 20947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:25:16.398396 20947 catalog_manager.cc:1259] Loaded cluster ID: 962145a11a604efe83ce42b46d1dbbdf
I20260923 17:25:16.398478 20947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:25:16.400195 20947 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:25:16.400810 20947 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1828ed9af465494eb3c18c3570bb5558: Loaded TSK: 0
I20260923 17:25:16.401978 20947 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 17:25:16.497362 20941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:16.497634 20941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:16.497702 20941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:16.501823 20941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:16.501890 20941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:16.501989 20941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.1
I20260923 17:25:16.507395 20941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:44481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.1.keytab
--principal=oryx/127.11.229.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.1:38951
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.229.1
--webserver_port=34867
--tserver_master_addrs=127.11.229.62:34307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.1
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:25:16.508533 20941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:16.509788 20941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:16.519476 20966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:16.519717 20941 server_base.cc:1034] running on GCE node
W20260923 17:25:16.519493 20968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:16.519493 20965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:16.520355 20941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:16.521085 20941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:16.522269 20941 hybrid_clock.cc:648] HybridClock initialized: now 1790184316522247 us; error 57 us; skew 500 ppm
Sep 23 17:25:16 dist-test-slave-ptl9 krb5kdc[20358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184316, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:16.525108 20941 init.cc:377] Logged in from keytab as oryx/127.11.229.1@KRBTEST.COM (short username oryx)
I20260923 17:25:16.526049 20941 webserver.cc:460] Webserver started at http://127.11.229.1:34867/ using document root <none> and password file <none>
I20260923 17:25:16.526628 20941 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:16.526697 20941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:16.530694 20941 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260923 17:25:16.533004 20975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:16.534787 20941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:25:16.534924 20941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-0/wal
uuid: "14eab5e51552440eaad8fec14bcbe48a"
format_stamp: "Formatted at 2026-09-23 17:25:15 on dist-test-slave-ptl9"
server_key: "e08926fc578c4cd61aecce6ed61d242f"
server_key_iv: "c38c590b8fa42bd907a491ad58ecba1f"
server_key_version: "encryptionkey@0"
I20260923 17:25:16.535535 20941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:16.559149 20941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:16.562670 20941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:16.562888 20941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:16.563585 20941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:25:16.564584 20941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:25:16.564648 20941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:16.564719 20941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:25:16.564749 20941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:16.574832 20941 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.1:38951
I20260923 17:25:16.574898 21088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.1:38951 every 8 connection(s)
I20260923 17:25:16.575903 20941 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
I20260923 17:25:16.584749 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 20941
I20260923 17:25:16.585210 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.2:40359
--local_ip_for_outbound_sockets=127.11.229.2
--tserver_master_addrs=127.11.229.62:34307
--webserver_port=41845
--webserver_interface=127.11.229.2
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:44481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.229.2
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:25:16 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:16 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:16.588403 21092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:16.578191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.1:53903)
0923 17:25:16.578873 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:16.578915 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:16.579677 (+ 762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:16.580726 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:16.580732 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:16.581075 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:16.581627 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:16.581642 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.582842 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.582844 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:16.583698 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:16.583707 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.584424 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.584426 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:16.584556 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:16.585220 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:16.585236 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:16.587683 (+ 2447us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":131,"threads_started":1}
W20260923 17:25:16.588817 21091 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:16.578019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:53903 (local address 127.11.229.62:34307)
0923 17:25:16.578485 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:16.578505 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:16.579039 (+ 534us) server_negotiation.cc:407] Connection header received
0923 17:25:16.580050 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:16.580060 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:16.580348 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:16.580621 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:16.581781 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.582728 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.583820 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.584281 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.584409 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:16.587947 (+ 3538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.1:53903: BlockingRecv error: recv got EOF from 127.11.229.1:53903 (error 108)
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":92,"threads_started":1}
W20260923 17:25:16.589366 21089 heartbeater.cc:644] Failed to heartbeat to 127.11.229.62:34307 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.229.62:34307: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Sep 23 17:25:16 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:16 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:16.596016 21092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:16.589822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.1:51777)
0923 17:25:16.589992 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:16.590011 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:16.590098 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:16.590342 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:16.590347 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:16.590548 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:16.590691 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:16.590701 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.591409 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.591415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:16.592457 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:16.592471 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.592935 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.592942 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:16.593069 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:16.593607 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:16.593627 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:16.595842 (+ 2215us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":77}
W20260923 17:25:16.596043 21091 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:16.589951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:51777 (local address 127.11.229.62:34307)
0923 17:25:16.590054 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:16.590057 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:16.590069 (+ 12us) server_negotiation.cc:407] Connection header received
0923 17:25:16.590189 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:16.590191 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:16.590228 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:16.590292 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:16.590810 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.591299 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.592588 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.592806 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.592885 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:16.595957 (+ 3072us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.1:51777: BlockingRecv error: recv got EOF from 127.11.229.1:51777 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 23 17:25:16 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:16 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:16.601574 21092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:16.596585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.1:45931)
0923 17:25:16.596846 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:16.596860 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:16.596964 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:16.597211 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:16.597213 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:16.597395 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:16.597468 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:16.597473 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.598178 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.598183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:16.598700 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:16.598708 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.599147 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.599151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:16.599229 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:16.599702 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:16.599723 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:16.601428 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195}
W20260923 17:25:16.601661 21091 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:16.596708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:45931 (local address 127.11.229.62:34307)
0923 17:25:16.596827 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:16.596831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:16.596927 (+ 96us) server_negotiation.cc:407] Connection header received
0923 17:25:16.597060 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:16.597063 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:16.597111 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:16.597195 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:16.597580 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.598066 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.598820 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.599033 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.599144 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:16.601543 (+ 2399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.1:45931: BlockingRecv error: recv got EOF from 127.11.229.1:45931 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260923 17:25:16.601892 21089 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 17:25:16.701687 21093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:16.702004 21093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:16.702098 21093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:16.706221 21093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:16.706305 21093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:16.706398 21093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.2
I20260923 17:25:16.711721 21093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:44481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.2.keytab
--principal=oryx/127.11.229.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.2:40359
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.229.2
--webserver_port=41845
--tserver_master_addrs=127.11.229.62:34307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.2
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:25:16.712890 21093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:16.714071 21093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:16.723888 21101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:16.723901 21099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:16.724193 21098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:16.724185 21093 server_base.cc:1034] running on GCE node
I20260923 17:25:16.724673 21093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:16.725374 21093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:16.726547 21093 hybrid_clock.cc:648] HybridClock initialized: now 1790184316726510 us; error 47 us; skew 500 ppm
Sep 23 17:25:16 dist-test-slave-ptl9 krb5kdc[20358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184316, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:16.729492 21093 init.cc:377] Logged in from keytab as oryx/127.11.229.2@KRBTEST.COM (short username oryx)
I20260923 17:25:16.730472 21093 webserver.cc:460] Webserver started at http://127.11.229.2:41845/ using document root <none> and password file <none>
I20260923 17:25:16.731076 21093 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:16.731151 21093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:16.735495 21093 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:25:16.737735 21108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:16.739020 21093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:25:16.739145 21093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-1/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-1/wal
uuid: "55920ce52d6241cbbb94986388a50e57"
format_stamp: "Formatted at 2026-09-23 17:25:15 on dist-test-slave-ptl9"
server_key: "144d6d2e12f12c22c817510674a87702"
server_key_iv: "8bad6cc726ddfa3ed7ba40376207b8a8"
server_key_version: "encryptionkey@0"
I20260923 17:25:16.739692 21093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:16.754561 21093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:16.757726 21093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:16.757949 21093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:16.758566 21093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:25:16.759454 21093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:25:16.759531 21093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:16.759603 21093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:25:16.759653 21093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:16.768874 21093 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.2:40359
I20260923 17:25:16.768898 21221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.2:40359 every 8 connection(s)
I20260923 17:25:16.769984 21093 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
I20260923 17:25:16.772353 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 21093
I20260923 17:25:16.772948 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.3:42203
--local_ip_for_outbound_sockets=127.11.229.3
--tserver_master_addrs=127.11.229.62:34307
--webserver_port=45171
--webserver_interface=127.11.229.3
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:44481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.229.3
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:25:16 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:16 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
W20260923 17:25:16.782289 21091 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:16.771894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:41365 (local address 127.11.229.62:34307)
0923 17:25:16.772048 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:16.772052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:16.773416 (+ 1364us) server_negotiation.cc:407] Connection header received
0923 17:25:16.774827 (+ 1411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:16.774830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:16.774897 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:16.775009 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:16.776226 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.776769 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.778043 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.778277 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.778393 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:16.782167 (+ 3774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.2:41365: BlockingRecv error: recv got EOF from 127.11.229.2:41365 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:25:16.783233 21224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:16.772312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.2:41365)
0923 17:25:16.773205 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:16.773290 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:16.774583 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:16.775209 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:16.775216 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:16.775570 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:16.776068 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:16.776081 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.776902 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.776905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:16.777900 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:16.777911 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.778438 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.778442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:16.778613 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:16.779265 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:16.779281 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:16.782028 (+ 2747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":139,"threads_started":1}
W20260923 17:25:16.783802 21222 heartbeater.cc:644] Failed to heartbeat to 127.11.229.62:34307 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.229.62:34307: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Sep 23 17:25:16 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:16 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:16.789572 21224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:16.784343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.2:50645)
0923 17:25:16.784447 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:16.784463 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:16.784548 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:16.784801 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:16.784804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:16.784972 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:16.785064 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:16.785069 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.785885 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.785889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:16.786449 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:16.786457 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.786932 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.786935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:16.787057 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:16.787571 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:16.787583 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:16.789437 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260923 17:25:16.790068 21091 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:16.784435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:50645 (local address 127.11.229.62:34307)
0923 17:25:16.784548 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:16.784553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:16.784568 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:25:16.784642 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:16.784645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:16.784696 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:16.784784 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:16.785190 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.785771 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.786573 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.786812 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.786928 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:16.789574 (+ 2646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.2:50645: BlockingRecv error: recv got EOF from 127.11.229.2:50645 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 23 17:25:16 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:16 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:16.795925 21224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:16.790182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.2:44217)
0923 17:25:16.790395 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:16.790410 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:16.790496 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:16.790733 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:16.790736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:16.790898 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:16.790991 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:16.790996 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.791694 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.791697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:16.792568 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:16.792580 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.793234 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.793237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:16.793384 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:16.793928 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:16.793939 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:16.795800 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145}
W20260923 17:25:16.796293 21222 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 17:25:16.796478 21091 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:16.790277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:44217 (local address 127.11.229.62:34307)
0923 17:25:16.790404 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:16.790408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:16.790454 (+ 46us) server_negotiation.cc:407] Connection header received
0923 17:25:16.790583 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:16.790585 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:16.790623 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:16.790690 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:16.791093 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.791584 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.792746 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.793118 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.793218 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:16.795933 (+ 2715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.2:44217: BlockingRecv error: recv got EOF from 127.11.229.2:44217 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260923 17:25:16.890033 21225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:16.890297 21225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:16.890370 21225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:16.894533 21225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:16.894596 21225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:16.894706 21225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.3
I20260923 17:25:16.900244 21225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:44481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/krb5kdc/oryx_127.11.229.3.keytab
--principal=oryx/127.11.229.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.3:42203
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.229.3
--webserver_port=45171
--tserver_master_addrs=127.11.229.62:34307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.3
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:25:16.901388 21225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:16.902415 21225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:16.912176 21231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:16.912302 21225 server_base.cc:1034] running on GCE node
W20260923 17:25:16.912194 21233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:16.912209 21230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:16.912858 21225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:16.913604 21225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:16.914798 21225 hybrid_clock.cc:648] HybridClock initialized: now 1790184316914762 us; error 48 us; skew 500 ppm
Sep 23 17:25:16 dist-test-slave-ptl9 krb5kdc[20358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184316, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.229.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:16.917624 21225 init.cc:377] Logged in from keytab as oryx/127.11.229.3@KRBTEST.COM (short username oryx)
I20260923 17:25:16.918435 21225 webserver.cc:460] Webserver started at http://127.11.229.3:45171/ using document root <none> and password file <none>
I20260923 17:25:16.918977 21225 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:16.919096 21225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:16.923275 21225 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:25:16.925539 21240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:16.926797 21225 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:25:16.926908 21225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-2/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-2/wal
uuid: "44b4389d5063459f928d788333ae2a7c"
format_stamp: "Formatted at 2026-09-23 17:25:16 on dist-test-slave-ptl9"
server_key: "1f06d836331f33f9826577021882a1b9"
server_key_iv: "1ab666be0deceaeea8586302ababb8d0"
server_key_version: "encryptionkey@0"
I20260923 17:25:16.927433 21225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:16.942399 21225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:16.945838 21225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:16.946004 21225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:16.946579 21225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:25:16.947373 21225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:25:16.947427 21225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:16.947467 21225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:25:16.947489 21225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:16.956313 21225 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.3:42203
I20260923 17:25:16.956333 21353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.3:42203 every 8 connection(s)
I20260923 17:25:16.957386 21225 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
I20260923 17:25:16.959918 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 21225
Sep 23 17:25:16 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.3@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:16 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.3@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
W20260923 17:25:16.970314 21091 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:16.959258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:55163 (local address 127.11.229.62:34307)
0923 17:25:16.959396 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:16.959400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:16.960747 (+ 1347us) server_negotiation.cc:407] Connection header received
0923 17:25:16.961966 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:16.961970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:16.962024 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:16.962143 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:16.963545 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.964391 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.965936 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.966335 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.966473 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:16.970175 (+ 3702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.3:55163: BlockingRecv error: recv got EOF from 127.11.229.3:55163 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 17:25:16.971038 21356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:16.959593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.3:55163)
0923 17:25:16.960592 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:16.960645 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:16.961747 (+ 1102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:16.962304 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:16.962310 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:16.962752 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:16.963385 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:16.963405 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.964508 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.964510 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:16.965818 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:16.965828 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.966508 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.966510 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:16.966693 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:16.967247 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:16.967269 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:16.970026 (+ 2757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":760,"thread_start_us":619,"threads_started":1}
I20260923 17:25:16.971058 21358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:16.961010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55848 (local address 127.11.229.62:34307)
0923 17:25:16.961342 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:16.961346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:16.961361 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:25:16.961407 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:16.961412 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:16.961463 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:16.961558 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:16.962582 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.963367 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.964631 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.964970 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.965076 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:16.966790 (+ 1714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:16.966798 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:16.966818 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:16.966854 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:16.968670 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:16.969185 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:16.969190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:16.969195 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:16.969372 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:16.969690 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:16.969693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:16.969694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:16.970072 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:16.970253 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:16.970730 (+ 477us) server_negotiation.cc:299] Negotiation successful
0923 17:25:16.970948 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":96,"threads_started":1}
W20260923 17:25:16.972190 21354 heartbeater.cc:644] Failed to heartbeat to 127.11.229.62:34307 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.229.62:34307: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Sep 23 17:25:16 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.3@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:16 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.3@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:16.977732 21356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:16.972655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.3:37493)
0923 17:25:16.972834 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:16.972849 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:16.972945 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:16.973375 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:16.973378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:16.973532 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:16.973619 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:16.973624 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.974356 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.974361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:16.974921 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:16.974929 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.975382 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.975386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:16.975484 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:16.975894 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:16.975905 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:16.977584 (+ 1679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102}
W20260923 17:25:16.977818 21358 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:16.972755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:37493 (local address 127.11.229.62:34307)
0923 17:25:16.972893 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:16.972898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:16.972912 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:25:16.973194 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:16.973197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:16.973233 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:16.973356 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:16.973743 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.974243 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.975044 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.975270 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.975382 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:16.977702 (+ 2320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.3:37493: BlockingRecv error: recv got EOF from 127.11.229.3:37493 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 23 17:25:16 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.3@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:16 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.3@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:16.982568 21356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:16.978329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.3:45311)
0923 17:25:16.978463 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:16.978479 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:16.978556 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:16.978747 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:16.978749 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:16.978865 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:16.978933 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:16.978937 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.979568 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.979570 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:16.980068 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:16.980074 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.980444 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.980446 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:16.980501 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:16.980887 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:16.980898 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:16.982476 (+ 1578us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":59}
W20260923 17:25:16.982739 21358 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:16.978412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:45311 (local address 127.11.229.62:34307)
0923 17:25:16.978561 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:16.978564 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:16.978575 (+ 11us) server_negotiation.cc:407] Connection header received
0923 17:25:16.978618 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:16.978620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:16.978660 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:16.978716 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:16.979023 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.979483 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.980161 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:16.980352 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:16.980422 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:16.982605 (+ 2183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.3:45311: BlockingRecv error: recv got EOF from 127.11.229.3:45311 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260923 17:25:16.982868 21354 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 23 17:25:17 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:17 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:17.607766 21359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:17.602541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.1:42239)
0923 17:25:17.602834 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:17.602848 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:17.602940 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:17.603214 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:17.603216 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:17.603393 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:17.603492 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:17.603496 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:17.604226 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:17.604228 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:17.604789 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:17.604797 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:17.605235 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:17.605238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:17.605326 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:17.605793 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:17.605804 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:17.607640 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":100,"threads_started":1}
W20260923 17:25:17.607946 21360 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:17.602619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:42239 (local address 127.11.229.62:34307)
0923 17:25:17.602895 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:17.602899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:17.602912 (+ 13us) server_negotiation.cc:407] Connection header received
0923 17:25:17.603032 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:17.603034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:17.603081 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:17.603177 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:17.603593 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:17.604120 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:17.604908 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:17.605138 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:17.605213 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:17.607810 (+ 2597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.1:42239: BlockingRecv error: recv got EOF from 127.11.229.1:42239 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":98,"threads_started":1}
Sep 23 17:25:17 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:17 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:17.803025 21361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:17.797003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.2:60181)
0923 17:25:17.797338 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:17.797351 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:17.797460 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:17.797700 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:17.797702 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:17.797877 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:17.797977 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:17.797981 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:17.798756 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:17.798759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:17.799343 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:17.799353 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:17.799836 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:17.799838 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:17.799904 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:17.800541 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:17.800553 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:17.802857 (+ 2304us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":79,"threads_started":1}
W20260923 17:25:17.803138 21360 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:17.797068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:60181 (local address 127.11.229.62:34307)
0923 17:25:17.797224 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:17.797228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:17.797421 (+ 193us) server_negotiation.cc:407] Connection header received
0923 17:25:17.797534 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:17.797537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:17.797592 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:17.797663 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:17.798084 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:17.798625 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:17.799482 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:17.799724 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:17.799815 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:17.803003 (+ 3188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.2:60181: BlockingRecv error: recv got EOF from 127.11.229.2:60181 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 23 17:25:17 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.3@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:17 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.3@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:17.988899 21362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:17.983539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.3:49085)
0923 17:25:17.983824 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:17.983837 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:17.983968 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:17.984234 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:17.984236 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:17.984443 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:17.984533 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:17.984539 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:17.985318 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:17.985322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:17.985897 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:17.985905 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:17.986367 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:17.986371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:17.986458 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:17.986942 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:17.986954 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:17.988774 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":82,"threads_started":1}
W20260923 17:25:17.989086 21360 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:17.983599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:49085 (local address 127.11.229.62:34307)
0923 17:25:17.983766 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:17.983771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:17.983910 (+ 139us) server_negotiation.cc:407] Connection header received
0923 17:25:17.984064 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:17.984067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:17.984125 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:17.984210 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:17.984652 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:17.985181 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:17.986015 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:17.986256 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:17.986362 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:17.988930 (+ 2568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.3:49085: BlockingRecv error: recv got EOF from 127.11.229.3:49085 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 23 17:25:18 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:18 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:18.614454 21363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:18.608607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.1:37197)
0923 17:25:18.608939 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:18.608953 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:18.609052 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:18.609323 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:18.609325 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:18.609496 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:18.609606 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:18.609611 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:18.610792 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:18.610795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:18.611357 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:18.611365 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:18.611829 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:18.611832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:18.611901 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:18.612410 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:18.612425 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:18.614315 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":91,"threads_started":1}
W20260923 17:25:18.614635 21364 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:18.608705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:37197 (local address 127.11.229.62:34307)
0923 17:25:18.608972 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:18.608976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:18.609003 (+ 27us) server_negotiation.cc:407] Connection header received
0923 17:25:18.609156 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:18.609158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:18.609205 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:18.609306 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:18.609747 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:18.610677 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:18.611470 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:18.611707 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:18.611785 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:18.614474 (+ 2689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.1:37197: BlockingRecv error: recv got EOF from 127.11.229.1:37197 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":74,"threads_started":1}
Sep 23 17:25:18 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:18 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:18.809108 21365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:18.803827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.2:39837)
0923 17:25:18.804144 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:18.804158 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:18.804260 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:18.804490 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:18.804493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:18.804672 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:18.804771 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:18.804776 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:18.805586 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:18.805588 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:18.806150 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:18.806158 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:18.806600 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:18.806603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:18.806667 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:18.807169 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:18.807180 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:18.808962 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":109,"threads_started":1}
W20260923 17:25:18.809250 21364 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:18.803877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:39837 (local address 127.11.229.62:34307)
0923 17:25:18.804029 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:18.804034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:18.804215 (+ 181us) server_negotiation.cc:407] Connection header received
0923 17:25:18.804339 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:18.804341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:18.804389 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:18.804450 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:18.804873 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:18.805495 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:18.806277 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:18.806498 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:18.806593 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:18.809123 (+ 2530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.2:39837: BlockingRecv error: recv got EOF from 127.11.229.2:39837 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 23 17:25:18 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.3@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:18 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.3@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:18.994870 21366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:18.989696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.3:58895)
0923 17:25:18.989954 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:18.989969 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:18.990071 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:18.990308 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:18.990309 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:18.990493 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:18.990589 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:18.990594 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:18.991300 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:18.991303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:18.991861 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:18.991868 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:18.992320 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:18.992322 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:18.992388 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:18.992834 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:18.992845 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:18.994722 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":81,"threads_started":1}
W20260923 17:25:18.994979 21364 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:18.989814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:58895 (local address 127.11.229.62:34307)
0923 17:25:18.989969 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:18.989973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:18.990037 (+ 64us) server_negotiation.cc:407] Connection header received
0923 17:25:18.990148 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:18.990150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:18.990197 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:18.990263 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:18.990681 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:18.991200 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:18.991955 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:18.992228 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:18.992297 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:18.994851 (+ 2554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.3:58895: BlockingRecv error: recv got EOF from 127.11.229.3:58895 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 23 17:25:19 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:19 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:19.621160 21367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:19.615340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.1:57531)
0923 17:25:19.615642 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:19.615656 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:19.615747 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:19.616016 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:19.616018 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:19.616222 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:19.616323 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:19.616328 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:19.617145 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:19.617148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:19.617858 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:19.617871 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:19.618521 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:19.618524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:19.618628 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:19.619216 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:19.619228 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:19.621023 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":109,"threads_started":1}
W20260923 17:25:19.621366 21368 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:19.615403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:57531 (local address 127.11.229.62:34307)
0923 17:25:19.615685 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:19.615688 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:19.615703 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:25:19.615841 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:19.615843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:19.615890 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:19.615985 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:19.616469 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:19.617034 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:19.618067 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:19.618403 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:19.618528 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:19.621175 (+ 2647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.1:57531: BlockingRecv error: recv got EOF from 127.11.229.1:57531 (error 108)
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":116,"threads_started":1}
Sep 23 17:25:19 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:19 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:19.815208 21369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:19.809995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.2:44009)
0923 17:25:19.810292 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:19.810305 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:19.810401 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:19.810615 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:19.810617 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:19.810798 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:19.810895 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:19.810900 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:19.811626 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:19.811628 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:19.812184 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:19.812191 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:19.812620 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:19.812623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:19.812684 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:19.813180 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:19.813192 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:19.815089 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":77,"threads_started":1}
W20260923 17:25:19.815469 21368 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:19.810064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:44009 (local address 127.11.229.62:34307)
0923 17:25:19.810185 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:19.810189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:19.810353 (+ 164us) server_negotiation.cc:407] Connection header received
0923 17:25:19.810467 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:19.810469 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:19.810511 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:19.810577 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:19.810995 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:19.811532 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:19.812292 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:19.812522 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:19.812599 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:19.815241 (+ 2642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.2:44009: BlockingRecv error: recv got EOF from 127.11.229.2:44009 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 23 17:25:19 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.3@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:19 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.3@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:20.001318 21370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:19.995616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.3:57341)
0923 17:25:19.995943 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:19.995963 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:19.996119 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:19.996397 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:19.996400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:19.996611 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:19.996712 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:19.996717 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:19.997539 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:19.997544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:19.998115 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:19.998123 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:19.998579 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:19.998583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:19.998672 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:19.999233 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:19.999247 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:20.001142 (+ 1895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":102,"threads_started":1}
W20260923 17:25:20.001590 21368 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:19.995738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:57341 (local address 127.11.229.62:34307)
0923 17:25:19.995899 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:19.995905 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:19.996037 (+ 132us) server_negotiation.cc:407] Connection header received
0923 17:25:19.996227 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:19.996230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:19.996289 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:19.996374 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:19.996833 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:19.997435 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:19.998237 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:19.998466 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:19.998585 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:20.001459 (+ 2874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.3:57341: BlockingRecv error: recv got EOF from 127.11.229.3:57341 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 23 17:25:20 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:20 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:20.632447 21372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:20.622226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.1:35583)
0923 17:25:20.622565 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:20.622582 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:20.622701 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:20.622984 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:20.622987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:20.623201 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:20.623295 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:20.623301 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:20.624076 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:20.624081 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:20.624651 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:20.624659 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:20.625137 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:20.625140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:20.625229 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:20.630332 (+ 5103us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:20.630345 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:20.632325 (+ 1980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":119,"threads_started":1}
W20260923 17:25:20.632606 21371 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:20.622264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:35583 (local address 127.11.229.62:34307)
0923 17:25:20.622565 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:20.622570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:20.622654 (+ 84us) server_negotiation.cc:407] Connection header received
0923 17:25:20.622802 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:20.622805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:20.622861 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:20.622957 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:20.623418 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:20.623966 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:20.624773 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:20.625017 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:20.625124 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:20.632472 (+ 7348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.1:35583: BlockingRecv error: recv got EOF from 127.11.229.1:35583 (error 108)
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":118,"threads_started":1}
Sep 23 17:25:20 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:20 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:20.822180 21373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:20.816105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.2:45547)
0923 17:25:20.816429 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:20.816451 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:20.816611 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:20.816913 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:20.816916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:20.817141 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:20.817237 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:20.817243 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:20.818057 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:20.818062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:20.818647 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:20.818655 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:20.819146 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:20.819150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:20.819241 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:20.819833 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:20.819848 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:20.822037 (+ 2189us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":90,"threads_started":1}
W20260923 17:25:20.822455 21371 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:20.816220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:45547 (local address 127.11.229.62:34307)
0923 17:25:20.816391 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:20.816396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:20.816543 (+ 147us) server_negotiation.cc:407] Connection header received
0923 17:25:20.816733 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:20.816736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:20.816795 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:20.816889 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:20.817397 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:20.817939 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:20.818777 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:20.819024 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:20.819139 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:20.822206 (+ 3067us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.2:45547: BlockingRecv error: recv got EOF from 127.11.229.2:45547 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 23 17:25:21 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.3@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:21 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.3@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:21.008142 21374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:21.002220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.3:60667)
0923 17:25:21.002559 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:21.002577 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:21.002733 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:21.003033 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:21.003036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:21.003252 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:21.003347 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:21.003352 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:21.004149 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:21.004154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:21.004720 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:21.004728 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:21.005200 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:21.005203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:21.005332 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:21.005891 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:21.005903 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:21.007967 (+ 2064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":89,"threads_started":1}
W20260923 17:25:21.008201 21371 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:21.002361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:60667 (local address 127.11.229.62:34307)
0923 17:25:21.002543 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:21.002548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:21.002677 (+ 129us) server_negotiation.cc:407] Connection header received
0923 17:25:21.002842 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:21.002845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:21.002903 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:21.003007 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:21.003475 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:21.004029 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:21.004846 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:21.005076 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:21.005191 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:21.008093 (+ 2902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.3:60667: BlockingRecv error: recv got EOF from 127.11.229.3:60667 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 23 17:25:21 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:21 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:21.638760 21375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:21.633386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.1:34531)
0923 17:25:21.633720 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:21.633734 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:21.633831 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:21.634077 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:21.634079 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:21.634273 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:21.634370 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:21.634375 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:21.635129 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:21.635132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:21.635694 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:21.635702 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:21.636122 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:21.636125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:21.636195 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:21.636672 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:21.636685 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:21.638574 (+ 1889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":109,"threads_started":1}
W20260923 17:25:21.638885 21376 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:21.633494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:34531 (local address 127.11.229.62:34307)
0923 17:25:21.633733 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:21.633737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:21.633784 (+ 47us) server_negotiation.cc:407] Connection header received
0923 17:25:21.633918 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:21.633920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:21.633964 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:21.634041 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:21.634468 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:21.635025 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:21.635797 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:21.636016 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:21.636112 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:21.638738 (+ 2626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.1:34531: BlockingRecv error: recv got EOF from 127.11.229.1:34531 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":63,"threads_started":1}
Sep 23 17:25:21 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:21 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:21.828613 21377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:21.823053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.2:40351)
0923 17:25:21.823358 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:21.823371 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:21.823466 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:21.823707 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:21.823709 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:21.823895 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:21.823989 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:21.823994 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:21.824743 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:21.824746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:21.825402 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:21.825412 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:21.825834 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:21.825837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:21.825898 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:21.826421 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:21.826433 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:21.828434 (+ 2001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":115,"threads_started":1}
W20260923 17:25:21.828691 21376 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:21.823205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:40351 (local address 127.11.229.62:34307)
0923 17:25:21.823361 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:21.823364 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:21.823480 (+ 116us) server_negotiation.cc:407] Connection header received
0923 17:25:21.823532 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:21.823534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:21.823594 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:21.823670 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:21.824099 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:21.824647 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:21.825517 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:21.825737 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:21.825816 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:21.828577 (+ 2761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.2:40351: BlockingRecv error: recv got EOF from 127.11.229.2:40351 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 23 17:25:22 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.3@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:22 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.3@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:22.014811 21378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:22.008994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.3:35871)
0923 17:25:22.009373 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:22.009391 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:22.009530 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:22.009803 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:22.009806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:22.010048 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:22.010141 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:22.010147 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.010923 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.010927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:22.011499 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:22.011506 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.011984 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.011988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:22.012081 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:22.012610 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:22.012623 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:22.014650 (+ 2027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":156,"threads_started":1}
W20260923 17:25:22.015034 21376 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:22.009184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:35871 (local address 127.11.229.62:34307)
0923 17:25:22.009373 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:22.009380 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:22.009481 (+ 101us) server_negotiation.cc:407] Connection header received
0923 17:25:22.009636 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:22.009639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:22.009699 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:22.009784 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:22.010262 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.010812 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.011623 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.011865 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.011973 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:22.014819 (+ 2846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.3:35871: BlockingRecv error: recv got EOF from 127.11.229.3:35871 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 23 17:25:22 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:22 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:22.645350 21379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:22.639767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.1:36307)
0923 17:25:22.640084 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:22.640103 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:22.640208 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:22.640448 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:22.640450 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:22.640644 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:22.640750 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:22.640755 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.641566 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.641569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:22.642136 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:22.642144 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.642566 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.642569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:22.642640 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:22.643144 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:22.643156 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:22.645171 (+ 2015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":84,"threads_started":1}
W20260923 17:25:22.645479 21380 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:22.639874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:36307 (local address 127.11.229.62:34307)
0923 17:25:22.640186 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:22.640189 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:22.640203 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:25:22.640291 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:22.640293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:22.640338 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:22.640405 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:22.640852 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.641467 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.642239 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.642462 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.642552 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:22.645336 (+ 2784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.1:36307: BlockingRecv error: recv got EOF from 127.11.229.1:36307 (error 108)
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":134,"threads_started":1}
Sep 23 17:25:22 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:22 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:22.835263 21381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:22.829523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.2:52137)
0923 17:25:22.829803 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:22.829815 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:22.829913 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:22.830185 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:22.830187 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:22.830366 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:22.830456 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:22.830460 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.831327 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.831329 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:22.831913 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:22.831922 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.832396 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.832400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:22.832495 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:22.833085 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:22.833098 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:22.835142 (+ 2044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":89,"threads_started":1}
W20260923 17:25:22.835517 21380 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:22.829597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:52137 (local address 127.11.229.62:34307)
0923 17:25:22.829757 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:22.829761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:22.829863 (+ 102us) server_negotiation.cc:407] Connection header received
0923 17:25:22.829983 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:22.829985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:22.830028 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:22.830101 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:22.830574 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.831221 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.832010 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:22.832233 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:22.832359 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:22.835282 (+ 2923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.2:52137: BlockingRecv error: recv got EOF from 127.11.229.2:52137 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 23 17:25:23 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.3@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:23 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.3@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:23.020817 21382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:23.015516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.3:36115)
0923 17:25:23.015788 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:23.015802 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:23.015904 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:23.016146 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:23.016149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:23.016338 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:23.016435 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:23.016441 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:23.017185 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:23.017188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:23.017775 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:23.017783 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:23.018199 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:23.018202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:23.018264 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:23.018772 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:23.018784 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:23.020630 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":69,"threads_started":1}
W20260923 17:25:23.020855 21380 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:23.015639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:36115 (local address 127.11.229.62:34307)
0923 17:25:23.015800 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:23.015809 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:23.015869 (+ 60us) server_negotiation.cc:407] Connection header received
0923 17:25:23.015984 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:23.015986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:23.016033 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:23.016101 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:23.016532 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:23.017086 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:23.017878 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:23.018098 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:23.018188 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:23.020755 (+ 2567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.3:36115: BlockingRecv error: recv got EOF from 127.11.229.3:36115 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 23 17:25:23 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:23 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:23.651618 21383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:23.646138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.1:51233)
0923 17:25:23.646474 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:23.646489 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:23.646595 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:23.646849 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:23.646851 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:23.647040 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:23.647144 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:23.647149 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:23.647881 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:23.647883 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:23.648582 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:23.648594 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:23.649025 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:23.649028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:23.649091 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:23.649624 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:23.649636 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:23.651498 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":71,"threads_started":1}
W20260923 17:25:23.651834 21384 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:23.646216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:51233 (local address 127.11.229.62:34307)
0923 17:25:23.646499 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:23.646503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:23.646561 (+ 58us) server_negotiation.cc:407] Connection header received
0923 17:25:23.646682 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:23.646684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:23.646732 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:23.646808 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:23.647260 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:23.647788 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:23.648694 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:23.648925 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:23.649005 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:23.651645 (+ 2640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.1:51233: BlockingRecv error: recv got EOF from 127.11.229.1:51233 (error 108)
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":93,"threads_started":1}
Sep 23 17:25:23 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:23 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:23.841948 21385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:23.836093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.2:33191)
0923 17:25:23.836553 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:23.836566 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:23.836709 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:23.836984 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:23.836987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:23.837213 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:23.837405 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:23.837412 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:23.838184 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:23.838190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:23.838772 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:23.838780 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:23.839260 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:23.839263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:23.839352 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:23.839890 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:23.839902 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:23.841763 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":94,"threads_started":1}
W20260923 17:25:23.842038 21384 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:23.836156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:33191 (local address 127.11.229.62:34307)
0923 17:25:23.836457 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:23.836462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:23.836650 (+ 188us) server_negotiation.cc:407] Connection header received
0923 17:25:23.836816 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:23.836821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:23.836870 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:23.836995 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:23.837527 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:23.838070 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:23.838893 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:23.839140 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:23.839252 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:23.841911 (+ 2659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.2:33191: BlockingRecv error: recv got EOF from 127.11.229.2:33191 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 23 17:25:24 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.3@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:24 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.3@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:24.027665 21386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:24.021674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.3:34937)
0923 17:25:24.022106 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:24.022121 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:24.022271 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:24.022595 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:24.022598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:24.022821 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:24.022921 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:24.022928 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:24.023786 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:24.023791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:24.024423 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:24.024431 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:24.024939 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:24.024953 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:24.025067 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:24.025613 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:24.025625 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:24.027489 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":89,"threads_started":1}
W20260923 17:25:24.027760 21384 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:24.021746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:34937 (local address 127.11.229.62:34307)
0923 17:25:24.021968 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:24.021974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:24.022217 (+ 243us) server_negotiation.cc:407] Connection header received
0923 17:25:24.022394 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:24.022399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:24.022466 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:24.022564 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:24.023062 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:24.023660 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:24.024557 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:24.024811 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:24.024933 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:24.027623 (+ 2690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.3:34937: BlockingRecv error: recv got EOF from 127.11.229.3:34937 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Sep 23 17:25:24 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:24 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:24.657613 21387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:24.652382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.1:56405)
0923 17:25:24.652677 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:24.652691 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:24.652783 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:24.653094 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:24.653096 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:24.653314 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:24.653416 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:24.653421 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:24.654165 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:24.654167 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:24.654744 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:24.654755 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:24.655171 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:24.655174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:24.655236 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:24.655718 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:24.655729 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:24.657493 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":105,"threads_started":1}
W20260923 17:25:24.657787 21388 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:24.652576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:56405 (local address 127.11.229.62:34307)
0923 17:25:24.652861 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:24.652865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:24.652878 (+ 13us) server_negotiation.cc:407] Connection header received
0923 17:25:24.652914 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:24.652916 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:24.652974 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:24.653051 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:24.653525 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:24.654074 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:24.654854 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:24.655077 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:24.655152 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:24.657640 (+ 2488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.1:56405: BlockingRecv error: recv got EOF from 127.11.229.1:56405 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":90,"threads_started":1}
Sep 23 17:25:24 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:24 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:24.848614 21389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:24.842809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.2:34489)
0923 17:25:24.843082 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:24.843095 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:24.843198 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:24.843440 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:24.843442 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:24.843617 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:24.843722 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:24.843728 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:24.844690 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:24.844692 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:24.845394 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:24.845405 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:24.845833 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:24.845836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:24.845898 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:24.846386 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:24.846397 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:24.848479 (+ 2082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":74,"threads_started":1}
W20260923 17:25:24.848680 21388 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:24.842897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:34489 (local address 127.11.229.62:34307)
0923 17:25:24.843108 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:24.843112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:24.843173 (+ 61us) server_negotiation.cc:407] Connection header received
0923 17:25:24.843284 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:24.843286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:24.843335 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:24.843405 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:24.843823 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:24.844583 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:24.845514 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:24.845733 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:24.845815 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:24.848580 (+ 2765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.2:34489: BlockingRecv error: recv got EOF from 127.11.229.2:34489 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 23 17:25:25 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.3@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:25 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.3@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:25.034101 21390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:25.028529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.3:55921)
0923 17:25:25.028800 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:25.028814 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:25.028903 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:25.029134 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:25.029137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:25.029342 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:25.029440 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:25.029446 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:25.030167 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:25.030170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:25.030723 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:25.030730 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:25.031185 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:25.031187 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:25.031252 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:25.031788 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:25.031801 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:25.033919 (+ 2118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":97,"threads_started":1}
W20260923 17:25:25.034214 21388 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:25.028727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:55921 (local address 127.11.229.62:34307)
0923 17:25:25.028874 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:25.028878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:25.028892 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:25:25.028975 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:25.028977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:25.029023 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:25.029094 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:25.029544 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:25.030072 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:25.030818 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:25.031072 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:25.031164 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:25.034083 (+ 2919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.3:55921: BlockingRecv error: recv got EOF from 127.11.229.3:55921 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 23 17:25:25 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:25 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:25.668784 21391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:25.658480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.1:60043)
0923 17:25:25.658781 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:25.658795 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:25.658890 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:25.659142 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:25.659144 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:25.659329 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:25.659434 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:25.659438 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:25.660164 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:25.660167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:25.660720 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:25.660727 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:25.661134 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:25.661137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:25.661198 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:25.666678 (+ 5480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:25.666692 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:25.668645 (+ 1953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":93,"threads_started":1}
W20260923 17:25:25.668973 21392 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:25.658549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:60043 (local address 127.11.229.62:34307)
0923 17:25:25.658821 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:25.658825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:25.658842 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:25:25.658979 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:25.658982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:25.659027 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:25.659100 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:25.659535 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:25.660077 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:25.660819 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:25.661040 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:25.661112 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:25.668797 (+ 7685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.1:60043: BlockingRecv error: recv got EOF from 127.11.229.1:60043 (error 108)
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":100,"threads_started":1}
Sep 23 17:25:25 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:25 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:25.855161 21393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:25.849572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.2:50213)
0923 17:25:25.849849 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:25.849863 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:25.849964 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:25.850215 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:25.850217 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:25.850399 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:25.850501 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:25.850506 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:25.851238 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:25.851240 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:25.851799 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:25.851806 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:25.852240 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:25.852243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:25.852305 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:25.852923 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:25.852947 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:25.854938 (+ 1991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":99,"threads_started":1}
W20260923 17:25:25.855173 21392 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:25.849660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:50213 (local address 127.11.229.62:34307)
0923 17:25:25.849841 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:25.849844 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:25.849937 (+ 93us) server_negotiation.cc:407] Connection header received
0923 17:25:25.850049 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:25.850052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:25.850099 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:25.850176 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:25.850610 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:25.851142 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:25.851899 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:25.852140 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:25.852217 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:25.855073 (+ 2856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.2:50213: BlockingRecv error: recv got EOF from 127.11.229.2:50213 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 23 17:25:26 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.3@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:26 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.3@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:26.040673 21394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:26.034895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.3:44803)
0923 17:25:26.035233 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:26.035246 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:26.035388 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:26.035663 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:26.035666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:26.035881 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:26.035978 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:26.035984 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:26.036785 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:26.036791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:26.037392 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:26.037401 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:26.037876 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:26.037880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:26.037968 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:26.038539 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:26.038552 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:26.040530 (+ 1978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":88,"threads_started":1}
W20260923 17:25:26.040843 21392 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:26.035010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:44803 (local address 127.11.229.62:34307)
0923 17:25:26.035182 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:26.035187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:26.035324 (+ 137us) server_negotiation.cc:407] Connection header received
0923 17:25:26.035492 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:26.035495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:26.035552 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:26.035640 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:26.036104 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:26.036656 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:26.037519 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:26.037760 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:26.037872 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:26.040684 (+ 2812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.3:44803: BlockingRecv error: recv got EOF from 127.11.229.3:44803 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 23 17:25:26 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:26 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:26.675166 21395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:26.669733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.1:37267)
0923 17:25:26.670012 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:26.670027 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:26.670114 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:26.670347 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:26.670349 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:26.670529 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:26.670642 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:26.670647 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:26.671486 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:26.671492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:26.672123 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:26.672133 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:26.672575 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:26.672578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:26.672650 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:26.673177 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:26.673189 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:26.675045 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":90,"threads_started":1}
W20260923 17:25:26.675354 21396 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:26.669801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:37267 (local address 127.11.229.62:34307)
0923 17:25:26.670103 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:26.670107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:26.670121 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:25:26.670200 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:26.670202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:26.670246 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:26.670313 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:26.670777 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:26.671327 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:26.672236 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:26.672470 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:26.672554 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:26.675212 (+ 2658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.1:37267: BlockingRecv error: recv got EOF from 127.11.229.1:37267 (error 108)
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":62,"threads_started":1}
Sep 23 17:25:26 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:26 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:26.861646 21397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:26.855990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.2:36037)
0923 17:25:26.856392 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:26.856405 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:26.856551 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:26.856854 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:26.856857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:26.857070 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:26.857175 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:26.857180 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:26.858002 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:26.858007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:26.858575 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:26.858583 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:26.859046 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:26.859050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:26.859144 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:26.859638 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:26.859650 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:26.861472 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":78,"threads_started":1}
W20260923 17:25:26.861742 21396 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:26.856047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:36037 (local address 127.11.229.62:34307)
0923 17:25:26.856228 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:26.856233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:26.856489 (+ 256us) server_negotiation.cc:407] Connection header received
0923 17:25:26.856669 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:26.856672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:26.856751 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:26.856829 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:26.857322 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:26.857889 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:26.858701 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:26.858932 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:26.859042 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:26.861605 (+ 2563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.2:36037: BlockingRecv error: recv got EOF from 127.11.229.2:36037 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 23 17:25:27 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.3@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:27 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.3@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:27.047442 21398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:27.041477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.3:56897)
0923 17:25:27.041774 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:27.041789 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:27.041877 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:27.042136 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:27.042138 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:27.042313 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:27.042412 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:27.042417 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:27.043138 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:27.043140 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:27.043801 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:27.043812 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:27.044402 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:27.044406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:27.044510 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:27.045111 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:27.045128 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:27.047264 (+ 2136us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":103,"threads_started":1}
W20260923 17:25:27.047572 21396 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:27.041655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:56897 (local address 127.11.229.62:34307)
0923 17:25:27.041809 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:27.041813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:27.041831 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:25:27.041965 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:27.041967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:27.042016 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:27.042088 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:27.042517 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:27.043045 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:27.043948 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:27.044294 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:27.044377 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:27.047442 (+ 3065us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.3:56897: BlockingRecv error: recv got EOF from 127.11.229.3:56897 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 23 17:25:27 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:27 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:27.682103 21399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:27.676102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.1:44433)
0923 17:25:27.676421 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:27.676437 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:27.676534 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:27.676810 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:27.676812 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:27.676994 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:27.677094 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:27.677100 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:27.677922 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:27.677925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:27.678484 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:27.678492 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:27.679055 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:27.679058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:27.679162 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:27.679814 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:27.679827 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:27.681932 (+ 2105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":92,"threads_started":1}
W20260923 17:25:27.682189 21400 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:27.676192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:44433 (local address 127.11.229.62:34307)
0923 17:25:27.676463 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:27.676467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:27.676490 (+ 23us) server_negotiation.cc:407] Connection header received
0923 17:25:27.676632 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:27.676635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:27.676691 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:27.676770 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:27.677239 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:27.677818 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:27.678649 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:27.678909 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:27.679049 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:27.682063 (+ 3014us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.1:44433: BlockingRecv error: recv got EOF from 127.11.229.1:44433 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":60,"threads_started":1}
Sep 23 17:25:27 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:27 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:27.867941 21401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:27.862487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.2:55949)
0923 17:25:27.862765 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:27.862782 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:27.862864 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:27.863104 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:27.863106 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:27.863282 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:27.863378 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:27.863383 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:27.864196 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:27.864199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:27.864740 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:27.864747 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:27.865223 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:27.865225 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:27.865356 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:27.865852 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:27.865863 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:27.867824 (+ 1961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":110,"threads_started":1}
W20260923 17:25:27.868124 21400 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:27.862617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:55949 (local address 127.11.229.62:34307)
0923 17:25:27.862766 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:27.862770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:27.862853 (+ 83us) server_negotiation.cc:407] Connection header received
0923 17:25:27.862950 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:27.862952 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:27.862998 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:27.863065 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:27.863483 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:27.864100 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:27.864850 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:27.865111 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:27.865197 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:27.867964 (+ 2767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.2:55949: BlockingRecv error: recv got EOF from 127.11.229.2:55949 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 23 17:25:28 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.3@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:28 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.3@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:28.053862 21402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:28.048327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.3:34003)
0923 17:25:28.048609 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:28.048623 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:28.048716 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:28.048966 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:28.048968 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:28.049150 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:28.049248 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:28.049287 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:28.050025 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:28.050027 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:28.050601 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:28.050609 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:28.051014 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:28.051017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:28.051077 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:28.051698 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:28.051710 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:28.053741 (+ 2031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":100,"threads_started":1}
W20260923 17:25:28.054176 21400 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:28.048512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:34003 (local address 127.11.229.62:34307)
0923 17:25:28.048645 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:28.048649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:28.048665 (+ 16us) server_negotiation.cc:407] Connection header received
0923 17:25:28.048807 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:28.048809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:28.048858 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:28.048927 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:28.049388 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:28.049924 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:28.050706 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:28.050919 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:28.050993 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:28.053890 (+ 2897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.3:34003: BlockingRecv error: recv got EOF from 127.11.229.3:34003 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 23 17:25:28 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:28 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:28.688633 21403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:28.682922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.1:59033)
0923 17:25:28.683229 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:28.683254 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:28.683354 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:28.683657 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:28.683659 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:28.683843 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:28.683946 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:28.683951 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:28.684815 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:28.684818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:28.685476 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:28.685486 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:28.685930 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:28.685933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:28.685998 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:28.686594 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:28.686606 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:28.688509 (+ 1903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":78,"threads_started":1}
W20260923 17:25:28.688858 21404 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:28.683075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:59033 (local address 127.11.229.62:34307)
0923 17:25:28.683322 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:28.683326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:28.683340 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:25:28.683438 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:28.683440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:28.683482 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:28.683559 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:28.684102 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:28.684687 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:28.685587 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:28.685818 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:28.685900 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:28.688698 (+ 2798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.1:59033: BlockingRecv error: recv got EOF from 127.11.229.1:59033 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":79,"threads_started":1}
Sep 23 17:25:28 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:28 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:28.874445 21405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:28.868734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.2:34313)
0923 17:25:28.869071 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:28.869087 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:28.869214 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:28.869552 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:28.869555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:28.869778 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:28.869873 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:28.869878 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:28.870659 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:28.870664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:28.871235 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:28.871243 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:28.871721 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:28.871725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:28.871813 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:28.872404 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:28.872415 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:28.874322 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":114,"threads_started":1}
W20260923 17:25:28.874617 21404 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:28.868895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:34313 (local address 127.11.229.62:34307)
0923 17:25:28.869064 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:28.869069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:28.869151 (+ 82us) server_negotiation.cc:407] Connection header received
0923 17:25:28.869374 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:28.869379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:28.869434 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:28.869530 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:28.869996 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:28.870538 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:28.871359 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:28.871601 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:28.871714 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:28.874461 (+ 2747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.2:34313: BlockingRecv error: recv got EOF from 127.11.229.2:34313 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 23 17:25:29 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.3@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:29 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.3@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:29.060107 21406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:29.054672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.3:52601)
0923 17:25:29.054954 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:29.054968 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:29.055071 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:29.055314 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:29.055316 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:29.055499 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:29.055596 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:29.055601 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:29.056340 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:29.056342 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:29.056890 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:29.056897 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:29.057390 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:29.057393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:29.057455 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:29.058025 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:29.058038 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:29.059989 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":93,"threads_started":1}
W20260923 17:25:29.060253 21404 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:29.054727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:52601 (local address 127.11.229.62:34307)
0923 17:25:29.054875 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:29.054880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:29.055043 (+ 163us) server_negotiation.cc:407] Connection header received
0923 17:25:29.055161 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:29.055163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:29.055212 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:29.055269 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:29.055720 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:29.056246 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:29.057001 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:29.057229 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:29.057336 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:29.060128 (+ 2792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.3:52601: BlockingRecv error: recv got EOF from 127.11.229.3:52601 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 23 17:25:29 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:29 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:29.695271 21407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:29.689573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.1:59813)
0923 17:25:29.689855 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:29.689872 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:29.689995 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:29.690334 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:29.690338 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:29.690556 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:29.690652 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:29.690657 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:29.691434 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:29.691439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:29.692029 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:29.692036 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:29.692511 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:29.692514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:29.692604 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:29.693130 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:29.693141 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:29.695143 (+ 2002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":98,"threads_started":1}
W20260923 17:25:29.695513 21408 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:29.689626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:59813 (local address 127.11.229.62:34307)
0923 17:25:29.689886 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:29.689890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:29.689953 (+ 63us) server_negotiation.cc:407] Connection header received
0923 17:25:29.690161 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:29.690165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:29.690221 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:29.690311 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:29.690774 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:29.691325 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:29.692152 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:29.692392 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:29.692496 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:29.695264 (+ 2768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.1:59813: BlockingRecv error: recv got EOF from 127.11.229.1:59813 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":76,"threads_started":1}
Sep 23 17:25:29 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:29 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:29.881078 21409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:29.875363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.2:39659)
0923 17:25:29.875741 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:29.875757 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:29.875908 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:29.876203 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:29.876206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:29.876418 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:29.876513 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:29.876519 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:29.877344 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:29.877348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:29.877933 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:29.877941 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:29.878437 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:29.878440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:29.878532 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:29.879018 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:29.879029 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:29.880888 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":91,"threads_started":1}
W20260923 17:25:29.881135 21408 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:29.875565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:39659 (local address 127.11.229.62:34307)
0923 17:25:29.875747 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:29.875753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:29.875845 (+ 92us) server_negotiation.cc:407] Connection header received
0923 17:25:29.876015 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:29.876018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:29.876070 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:29.876172 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:29.876640 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:29.877192 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:29.878059 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:29.878315 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:29.878426 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:29.881020 (+ 2594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.2:39659: BlockingRecv error: recv got EOF from 127.11.229.2:39659 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 23 17:25:30 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.3@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:30 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.3@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:30.066859 21410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:30.061130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.3:52295)
0923 17:25:30.061408 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:30.061433 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:30.061549 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:30.061799 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:30.061801 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:30.062054 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:30.062161 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:30.062166 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:30.063013 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:30.063017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:30.063746 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:30.063756 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:30.064242 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:30.064245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:30.064329 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:30.064817 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:30.064830 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:30.066741 (+ 1911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":128,"threads_started":1}
W20260923 17:25:30.067205 21408 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:30.061186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:52295 (local address 127.11.229.62:34307)
0923 17:25:30.061363 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:30.061366 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:30.061507 (+ 141us) server_negotiation.cc:407] Connection header received
0923 17:25:30.061636 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:30.061638 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:30.061692 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:30.061768 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:30.062279 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:30.062830 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:30.063858 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:30.064107 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:30.064219 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:30.067084 (+ 2865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.3:52295: BlockingRecv error: recv got EOF from 127.11.229.3:52295 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 23 17:25:30 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:30 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:30.705207 21411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:30.696147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.1:58989)
0923 17:25:30.696441 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:30.696459 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:30.696570 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:30.696874 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:30.696877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:30.697072 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:30.697175 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:30.697181 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:30.697932 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:30.697934 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:30.698503 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:30.698510 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:30.698933 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:30.698936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:30.698996 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:30.703092 (+ 4096us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:30.703105 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:30.705013 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":84,"threads_started":1}
W20260923 17:25:30.705448 21412 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:30.696201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:58989 (local address 127.11.229.62:34307)
0923 17:25:30.696533 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:30.696538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:30.696554 (+ 16us) server_negotiation.cc:407] Connection header received
0923 17:25:30.696662 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:30.696667 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:30.696723 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:30.696824 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:30.697300 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:30.697836 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:30.698608 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:30.698833 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:30.698910 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:30.705161 (+ 6251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.1:58989: BlockingRecv error: recv got EOF from 127.11.229.1:58989 (error 108)
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":60,"threads_started":1}
Sep 23 17:25:30 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:30 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:30.887969 21413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:30.881993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.2:50517)
0923 17:25:30.882405 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:30.882419 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:30.882571 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:30.882882 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:30.882885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:30.883106 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:30.883221 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:30.883227 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:30.884038 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:30.884045 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:30.884644 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:30.884652 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:30.885136 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:30.885139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:30.885229 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:30.885842 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:30.885855 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:30.887777 (+ 1922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":124,"threads_started":1}
W20260923 17:25:30.888027 21412 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:30.882148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:50517 (local address 127.11.229.62:34307)
0923 17:25:30.882300 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:30.882304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:30.882510 (+ 206us) server_negotiation.cc:407] Connection header received
0923 17:25:30.882695 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:30.882698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:30.882758 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:30.882854 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:30.883352 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:30.883914 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:30.884771 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:30.885005 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:30.885123 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:30.887912 (+ 2789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.2:50517: BlockingRecv error: recv got EOF from 127.11.229.2:50517 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 23 17:25:31 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.3@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:31 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.3@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:31.073889 21414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:31.067686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.3:36825)
0923 17:25:31.067989 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:31.068018 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:31.068151 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:31.068447 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:31.068450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:31.068671 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:31.068767 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:31.068773 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:31.069690 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:31.069695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:31.070304 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:31.070313 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:31.070886 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:31.070890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:31.070986 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:31.071647 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:31.071665 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:31.073765 (+ 2100us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":94,"threads_started":1}
W20260923 17:25:31.074086 21412 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:31.067755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:36825 (local address 127.11.229.62:34307)
0923 17:25:31.067935 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:31.067940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:31.068092 (+ 152us) server_negotiation.cc:407] Connection header received
0923 17:25:31.068283 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:31.068285 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:31.068334 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:31.068421 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:31.068887 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:31.069584 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:31.070448 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:31.070759 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:31.070865 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:31.073911 (+ 3046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.3:36825: BlockingRecv error: recv got EOF from 127.11.229.3:36825 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 23 17:25:31 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:31 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.1@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:31.711565 21415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:31.706237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.1:44521)
0923 17:25:31.706522 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:31.706536 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:31.706624 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:31.706864 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:31.706865 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:31.707044 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:31.707141 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:31.707146 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:31.707894 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:31.707897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:31.708450 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:31.708457 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:31.708881 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:31.708884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:31.708944 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:31.709568 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:31.709580 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:31.711445 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":75,"threads_started":1}
W20260923 17:25:31.711787 21416 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:31.706291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:44521 (local address 127.11.229.62:34307)
0923 17:25:31.706585 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:31.706588 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:31.706602 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:25:31.706728 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:31.706731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:31.706777 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:31.706852 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:31.707239 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:31.707797 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:31.708550 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:31.708784 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:31.708859 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:31.711595 (+ 2736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.1:44521: BlockingRecv error: recv got EOF from 127.11.229.1:44521 (error 108)
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":91,"threads_started":1}
Sep 23 17:25:31 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:25:31 dist-test-slave-ptl9 krb5kdc[20358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.229.2@KRBTEST.COM for oryx/127.11.229.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:25:31.895083 21417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:31.889072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34307 (local address 127.11.229.2:60033)
0923 17:25:31.889394 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:31.889407 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:31.889505 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:31.889792 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:31.889795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:31.890023 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:31.890125 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:31.890131 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:31.890942 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:31.890946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:31.891539 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:31.891548 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:31.892133 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:31.892136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:31.892255 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:31.892856 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:31.892877 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:31.894967 (+ 2090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.229.62:34307: Server oryx/127.11.229.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":115,"threads_started":1}
W20260923 17:25:31.895241 21416 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:31.889186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:60033 (local address 127.11.229.62:34307)
0923 17:25:31.889388 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:31.889392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:31.889477 (+ 85us) server_negotiation.cc:407] Connection header received
0923 17:25:31.889589 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:31.889591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:31.889636 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:31.889714 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:31.890249 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:31.890805 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:31.891687 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:31.892005 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:31.892114 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:31.895107 (+ 2993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.229.2:60033: BlockingRecv error: recv got EOF from 127.11.229.2:60033 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260923 17:25:31.960372 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 20941
I20260923 17:25:31.966404 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 21093
I20260923 17:25:31.971933 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 21225
I20260923 17:25:31.977094 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 20871
2026-09-23T17:25:31Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18036 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-23T17:25:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:25:31Z Disabled control of system clock
I20260923 17:25:32.008347 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:38573
--webserver_interface=127.11.229.62
--webserver_port=0
--builtin_ntp_servers=127.11.229.20:45663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:38573
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 17:25:32.122531 21421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:32.122789 21421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:32.122826 21421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:32.126861 21421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:25:32.126919 21421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:32.126933 21421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:32.126950 21421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:25:32.126965 21421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:25:32.131599 21421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:45663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:38573
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:38573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:25:32.132614 21421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:32.134095 21421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:32.142947 21426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:32.142966 21427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:32.143005 21421 server_base.cc:1034] running on GCE node
W20260923 17:25:32.142957 21429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:32.143877 21421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:32.145040 21421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:32.146204 21421 hybrid_clock.cc:648] HybridClock initialized: now 1790184332146167 us; error 59 us; skew 500 ppm
I20260923 17:25:32.148078 21421 webserver.cc:460] Webserver started at http://127.11.229.62:34545/ using document root <none> and password file <none>
I20260923 17:25:32.148669 21421 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:32.148730 21421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:32.148948 21421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:25:32.151145 21421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "d9af2e1ca50441e9b918bef6c050905f"
format_stamp: "Formatted at 2026-09-23 17:25:32 on dist-test-slave-ptl9"
server_key: "1af40be2c04eb6f6977558bc182edf6c"
server_key_iv: "207890d4a20d4b61f129b0ac79f64870"
server_key_version: "encryptionkey@0"
I20260923 17:25:32.151744 21421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "d9af2e1ca50441e9b918bef6c050905f"
format_stamp: "Formatted at 2026-09-23 17:25:32 on dist-test-slave-ptl9"
server_key: "1af40be2c04eb6f6977558bc182edf6c"
server_key_iv: "207890d4a20d4b61f129b0ac79f64870"
server_key_version: "encryptionkey@0"
I20260923 17:25:32.155614 21421 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:25:32.158032 21435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:32.159199 21421 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:25:32.159327 21421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "d9af2e1ca50441e9b918bef6c050905f"
format_stamp: "Formatted at 2026-09-23 17:25:32 on dist-test-slave-ptl9"
server_key: "1af40be2c04eb6f6977558bc182edf6c"
server_key_iv: "207890d4a20d4b61f129b0ac79f64870"
server_key_version: "encryptionkey@0"
I20260923 17:25:32.159437 21421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:32.174957 21421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:32.175654 21421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:32.175855 21421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:32.182946 21421 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:38573
I20260923 17:25:32.183041 21487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:38573 every 8 connection(s)
I20260923 17:25:32.183936 21421 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:25:32.184271 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 21421
I20260923 17:25:32.184369 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:25:32.184748 12180 external_mini_cluster.cc:1428] Setting key 30de21c8ea649cdcbd5f72963204f546
I20260923 17:25:32.187799 21488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:32.194871 21488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9af2e1ca50441e9b918bef6c050905f: Bootstrap starting.
I20260923 17:25:32.195650 21491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:32.186223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37746 (local address 127.11.229.62:38573)
0923 17:25:32.186903 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:32.186923 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:32.186965 (+ 42us) server_negotiation.cc:407] Connection header received
0923 17:25:32.187854 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:32.187879 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:32.188371 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:32.188746 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:32.189174 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:32.190710 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:32.191614 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:32.192306 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:32.192475 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:32.193445 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:32.193452 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:32.193469 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:25:32.194010 (+ 541us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:32.194084 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:25:32.194111 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:32.194210 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:32.194347 (+ 137us) server_negotiation.cc:299] Negotiation successful
0923 17:25:32.194637 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":120,"threads_started":1}
I20260923 17:25:32.197561 21488 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9af2e1ca50441e9b918bef6c050905f: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:32.198434 21488 log.cc:824] T 00000000000000000000000000000000 P d9af2e1ca50441e9b918bef6c050905f: Log is configured to *not* fsync() on all Append() calls
I20260923 17:25:32.200465 21488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9af2e1ca50441e9b918bef6c050905f: No bootstrap required, opened a new log
I20260923 17:25:32.204131 21488 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9af2e1ca50441e9b918bef6c050905f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9af2e1ca50441e9b918bef6c050905f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 38573 } }
I20260923 17:25:32.204411 21488 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9af2e1ca50441e9b918bef6c050905f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:32.204491 21488 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9af2e1ca50441e9b918bef6c050905f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9af2e1ca50441e9b918bef6c050905f, State: Initialized, Role: FOLLOWER
I20260923 17:25:32.204950 21488 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9af2e1ca50441e9b918bef6c050905f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9af2e1ca50441e9b918bef6c050905f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 38573 } }
I20260923 17:25:32.205104 21488 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9af2e1ca50441e9b918bef6c050905f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:25:32.205168 21488 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9af2e1ca50441e9b918bef6c050905f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:25:32.205312 21488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9af2e1ca50441e9b918bef6c050905f [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:32.206405 21488 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9af2e1ca50441e9b918bef6c050905f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9af2e1ca50441e9b918bef6c050905f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 38573 } }
I20260923 17:25:32.206703 21488 leader_election.cc:302] T 00000000000000000000000000000000 P d9af2e1ca50441e9b918bef6c050905f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9af2e1ca50441e9b918bef6c050905f; no voters:
I20260923 17:25:32.206977 21488 leader_election.cc:288] T 00000000000000000000000000000000 P d9af2e1ca50441e9b918bef6c050905f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:25:32.207115 21493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9af2e1ca50441e9b918bef6c050905f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:25:32.207368 21493 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9af2e1ca50441e9b918bef6c050905f [term 1 LEADER]: Becoming Leader. State: Replica: d9af2e1ca50441e9b918bef6c050905f, State: Running, Role: LEADER
I20260923 17:25:32.207763 21493 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9af2e1ca50441e9b918bef6c050905f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9af2e1ca50441e9b918bef6c050905f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 38573 } }
I20260923 17:25:32.208315 21488 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9af2e1ca50441e9b918bef6c050905f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:25:32.209476 21494 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9af2e1ca50441e9b918bef6c050905f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9af2e1ca50441e9b918bef6c050905f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9af2e1ca50441e9b918bef6c050905f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 38573 } } }
I20260923 17:25:32.209609 21494 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9af2e1ca50441e9b918bef6c050905f [sys.catalog]: This master's current role is: LEADER
I20260923 17:25:32.209894 21495 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9af2e1ca50441e9b918bef6c050905f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9af2e1ca50441e9b918bef6c050905f. Latest consensus state: current_term: 1 leader_uuid: "d9af2e1ca50441e9b918bef6c050905f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9af2e1ca50441e9b918bef6c050905f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 38573 } } }
I20260923 17:25:32.210011 21502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:25:32.210050 21495 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9af2e1ca50441e9b918bef6c050905f [sys.catalog]: This master's current role is: LEADER
I20260923 17:25:32.213336 21502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:25:32.218816 21502 catalog_manager.cc:1347] Generated new cluster ID: ee7e9f2cd08f4435bc1548c63f519207
I20260923 17:25:32.218890 21502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:25:32.229079 21502 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:25:32.230336 21502 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:25:32.236258 21502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9af2e1ca50441e9b918bef6c050905f: Generated new TSK 0
I20260923 17:25:32.237037 21502 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:25:32.248512 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.1:0
--local_ip_for_outbound_sockets=127.11.229.1
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:38573
--builtin_ntp_servers=127.11.229.20:45663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260923 17:25:32.364705 21512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:32.365003 21512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:32.365070 21512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:32.369354 21512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:32.369441 21512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:32.369532 21512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.1
I20260923 17:25:32.374446 21512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:45663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:38573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.1
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:25:32.375588 21512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:32.377045 21512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:32.386933 21517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:32.386933 21520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:32.386945 21518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:32.387265 21512 server_base.cc:1034] running on GCE node
I20260923 17:25:32.387679 21512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:32.388405 21512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:32.389595 21512 hybrid_clock.cc:648] HybridClock initialized: now 1790184332389555 us; error 54 us; skew 500 ppm
I20260923 17:25:32.391431 21512 webserver.cc:460] Webserver started at http://127.11.229.1:42951/ using document root <none> and password file <none>
I20260923 17:25:32.392052 21512 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:32.392149 21512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:32.392400 21512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:25:32.394614 21512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/data/instance:
uuid: "6344249e1b8143b989fb85b0578c6193"
format_stamp: "Formatted at 2026-09-23 17:25:32 on dist-test-slave-ptl9"
server_key: "78261db8c16a8eb3361a784b58163299"
server_key_iv: "f0e6954c633dbcb0b758ee7ed41476be"
server_key_version: "encryptionkey@0"
I20260923 17:25:32.395301 21512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance:
uuid: "6344249e1b8143b989fb85b0578c6193"
format_stamp: "Formatted at 2026-09-23 17:25:32 on dist-test-slave-ptl9"
server_key: "78261db8c16a8eb3361a784b58163299"
server_key_iv: "f0e6954c633dbcb0b758ee7ed41476be"
server_key_version: "encryptionkey@0"
I20260923 17:25:32.399221 21512 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:25:32.401933 21526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:32.403168 21512 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:25:32.403319 21512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/wal
uuid: "6344249e1b8143b989fb85b0578c6193"
format_stamp: "Formatted at 2026-09-23 17:25:32 on dist-test-slave-ptl9"
server_key: "78261db8c16a8eb3361a784b58163299"
server_key_iv: "f0e6954c633dbcb0b758ee7ed41476be"
server_key_version: "encryptionkey@0"
I20260923 17:25:32.403442 21512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:32.415457 21512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:32.416204 21512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:32.416410 21512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:32.417033 21512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:25:32.418052 21512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:25:32.418128 21512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:32.418195 21512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:25:32.418248 21512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:32.427330 21512 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.1:46845
I20260923 17:25:32.427380 21639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.1:46845 every 8 connection(s)
I20260923 17:25:32.428417 21512 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
I20260923 17:25:32.435145 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 21512
I20260923 17:25:32.435281 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance
I20260923 17:25:32.435611 12180 external_mini_cluster.cc:1428] Setting key 520c3792eb40a4991c305261723c18b3
I20260923 17:25:32.437690 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.2:0
--local_ip_for_outbound_sockets=127.11.229.2
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:38573
--builtin_ntp_servers=127.11.229.20:45663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260923 17:25:32.438745 21491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:32.430401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:34931 (local address 127.11.229.62:38573)
0923 17:25:32.430546 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:32.430550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:32.431649 (+ 1099us) server_negotiation.cc:407] Connection header received
0923 17:25:32.433106 (+ 1457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:32.433109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:32.433175 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:32.433295 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:32.434253 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:32.434804 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:32.436537 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:32.436763 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:32.436881 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:32.438241 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:32.438244 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:32.438246 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:25:32.438258 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:32.438278 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:25:32.438285 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:32.438348 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:32.438527 (+ 179us) server_negotiation.cc:299] Negotiation successful
0923 17:25:32.438604 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 17:25:32.439777 21642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:32.430815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:38573 (local address 127.11.229.1:34931)
0923 17:25:32.431493 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:32.431534 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:32.432875 (+ 1341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:32.433514 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:32.433521 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:32.433605 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:32.434091 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:32.434104 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:32.434995 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:32.434999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:32.436407 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:32.436421 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:32.436955 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:32.436960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:32.437180 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:32.437925 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:25:32.437929 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:32.437987 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:25:32.438073 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:32.438375 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:32.438386 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:32.438395 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 17:25:32.438532 (+ 137us) client_negotiation.cc:240] Negotiation successful
0923 17:25:32.438851 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":125,"threads_started":1}
I20260923 17:25:32.441133 21640 heartbeater.cc:347] Connected to a master server at 127.11.229.62:38573
I20260923 17:25:32.441438 21640 heartbeater.cc:464] Registering TS with master...
I20260923 17:25:32.442166 21640 heartbeater.cc:511] Master 127.11.229.62:38573 requested a full tablet report, sending...
I20260923 17:25:32.443882 21452 ts_manager.cc:194] Registered new tserver with Master: 6344249e1b8143b989fb85b0578c6193 (127.11.229.1:46845)
I20260923 17:25:32.445083 21452 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.229.1:34931
W20260923 17:25:32.557608 21643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:32.557891 21643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:32.557973 21643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:32.561992 21643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:32.562040 21643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:32.562127 21643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.2
I20260923 17:25:32.566987 21643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:45663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:38573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.2
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:25:32.568060 21643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:32.569694 21643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:32.579600 21651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:32.579594 21648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:32.579594 21649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:32.579931 21643 server_base.cc:1034] running on GCE node
I20260923 17:25:32.580375 21643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:32.581081 21643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:32.582265 21643 hybrid_clock.cc:648] HybridClock initialized: now 1790184332582256 us; error 57 us; skew 500 ppm
I20260923 17:25:32.584259 21643 webserver.cc:460] Webserver started at http://127.11.229.2:35485/ using document root <none> and password file <none>
I20260923 17:25:32.584862 21643 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:32.585011 21643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:32.585244 21643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:25:32.587348 21643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/data/instance:
uuid: "bd011f88b8da408289e60b32915d5a4a"
format_stamp: "Formatted at 2026-09-23 17:25:32 on dist-test-slave-ptl9"
server_key: "08647f441424d01e2eda508c2f19c587"
server_key_iv: "ef7d62e09cad227aed1866ee4dfd5aa2"
server_key_version: "encryptionkey@0"
I20260923 17:25:32.587950 21643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance:
uuid: "bd011f88b8da408289e60b32915d5a4a"
format_stamp: "Formatted at 2026-09-23 17:25:32 on dist-test-slave-ptl9"
server_key: "08647f441424d01e2eda508c2f19c587"
server_key_iv: "ef7d62e09cad227aed1866ee4dfd5aa2"
server_key_version: "encryptionkey@0"
I20260923 17:25:32.591863 21643 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260923 17:25:32.594811 21657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:32.596081 21643 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:25:32.596246 21643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/wal
uuid: "bd011f88b8da408289e60b32915d5a4a"
format_stamp: "Formatted at 2026-09-23 17:25:32 on dist-test-slave-ptl9"
server_key: "08647f441424d01e2eda508c2f19c587"
server_key_iv: "ef7d62e09cad227aed1866ee4dfd5aa2"
server_key_version: "encryptionkey@0"
I20260923 17:25:32.596367 21643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:32.619514 21643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:32.620286 21643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:32.620493 21643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:32.621129 21643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:25:32.622164 21643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:25:32.622232 21643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:32.622308 21643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:25:32.622349 21643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:32.630959 21643 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.2:43111
I20260923 17:25:32.631024 21770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.2:43111 every 8 connection(s)
I20260923 17:25:32.631892 21643 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
I20260923 17:25:32.634713 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 21643
I20260923 17:25:32.634831 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance
I20260923 17:25:32.635175 12180 external_mini_cluster.cc:1428] Setting key 224e556e3e0efa3404f07aa60533efad
I20260923 17:25:32.637308 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.3:0
--local_ip_for_outbound_sockets=127.11.229.3
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:38573
--builtin_ntp_servers=127.11.229.20:45663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260923 17:25:32.641812 21491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:32.633808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:48301 (local address 127.11.229.62:38573)
0923 17:25:32.633951 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:32.633955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:32.635205 (+ 1250us) server_negotiation.cc:407] Connection header received
0923 17:25:32.636706 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:32.636708 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:32.636751 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:32.636815 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:32.638107 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:32.638619 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:32.639791 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:32.640016 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:32.640132 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:32.641292 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:32.641297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:32.641304 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:25:32.641317 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:32.641338 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:25:32.641346 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:32.641420 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:32.641588 (+ 168us) server_negotiation.cc:299] Negotiation successful
0923 17:25:32.641661 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 17:25:32.642681 21773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:32.634217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:38573 (local address 127.11.229.2:48301)
0923 17:25:32.635034 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:32.635091 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:32.636479 (+ 1388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:32.637010 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:32.637021 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:32.637139 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:32.637943 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:32.637967 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:32.638754 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:32.638757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:32.639664 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:32.639674 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:32.640175 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:32.640179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:32.640349 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:32.640997 (+ 648us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:25:32.640999 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:32.641042 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:25:32.641115 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:32.641435 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:32.641449 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:32.641454 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 17:25:32.641588 (+ 134us) client_negotiation.cc:240] Negotiation successful
0923 17:25:32.641890 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":106,"threads_started":1}
I20260923 17:25:32.643959 21771 heartbeater.cc:347] Connected to a master server at 127.11.229.62:38573
I20260923 17:25:32.644214 21771 heartbeater.cc:464] Registering TS with master...
I20260923 17:25:32.644723 21771 heartbeater.cc:511] Master 127.11.229.62:38573 requested a full tablet report, sending...
I20260923 17:25:32.645784 21452 ts_manager.cc:194] Registered new tserver with Master: bd011f88b8da408289e60b32915d5a4a (127.11.229.2:43111)
I20260923 17:25:32.646345 21452 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.229.2:48301
W20260923 17:25:32.750178 21774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:32.750459 21774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:32.750521 21774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:32.754565 21774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:32.754637 21774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:32.754726 21774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.3
I20260923 17:25:32.759395 21774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:45663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:38573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.3
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:25:32.760504 21774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:32.761960 21774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 17:25:32.771562 21774 server_base.cc:1034] running on GCE node
W20260923 17:25:32.771498 21782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:32.771504 21779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:32.771498 21780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:32.772161 21774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:32.772859 21774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:32.774045 21774 hybrid_clock.cc:648] HybridClock initialized: now 1790184332774028 us; error 28 us; skew 500 ppm
I20260923 17:25:32.776003 21774 webserver.cc:460] Webserver started at http://127.11.229.3:39915/ using document root <none> and password file <none>
I20260923 17:25:32.776594 21774 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:32.776693 21774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:32.776927 21774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:25:32.779065 21774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/data/instance:
uuid: "0780efe01db64561a5199fa31be8dbcd"
format_stamp: "Formatted at 2026-09-23 17:25:32 on dist-test-slave-ptl9"
server_key: "98b9b432ac61bb295570a2db6bb60b31"
server_key_iv: "774b2ddc4d1d952263dccc5b7957a1f1"
server_key_version: "encryptionkey@0"
I20260923 17:25:32.779619 21774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance:
uuid: "0780efe01db64561a5199fa31be8dbcd"
format_stamp: "Formatted at 2026-09-23 17:25:32 on dist-test-slave-ptl9"
server_key: "98b9b432ac61bb295570a2db6bb60b31"
server_key_iv: "774b2ddc4d1d952263dccc5b7957a1f1"
server_key_version: "encryptionkey@0"
I20260923 17:25:32.783363 21774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 17:25:32.785881 21788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:32.787067 21774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:25:32.787202 21774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/wal
uuid: "0780efe01db64561a5199fa31be8dbcd"
format_stamp: "Formatted at 2026-09-23 17:25:32 on dist-test-slave-ptl9"
server_key: "98b9b432ac61bb295570a2db6bb60b31"
server_key_iv: "774b2ddc4d1d952263dccc5b7957a1f1"
server_key_version: "encryptionkey@0"
I20260923 17:25:32.787314 21774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:32.819636 21774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:32.820438 21774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:32.820647 21774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:32.821359 21774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:25:32.822384 21774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:25:32.822454 21774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:32.822525 21774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:25:32.822569 21774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:32.831712 21774 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.3:36555
I20260923 17:25:32.831732 21901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.3:36555 every 8 connection(s)
I20260923 17:25:32.832758 21774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
I20260923 17:25:32.833750 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 21774
I20260923 17:25:32.833837 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance
I20260923 17:25:32.834087 12180 external_mini_cluster.cc:1428] Setting key b2939e18864b91037f5a88f1419c211b
I20260923 17:25:32.843377 21491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:32.835674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:35041 (local address 127.11.229.62:38573)
0923 17:25:32.835804 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:32.835808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:32.837153 (+ 1345us) server_negotiation.cc:407] Connection header received
0923 17:25:32.838631 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:32.838634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:32.838690 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:32.838788 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:32.839705 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:32.840242 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:32.841449 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:32.841677 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:32.841788 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:32.842881 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:32.842884 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:32.842887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:25:32.842905 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:32.842924 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:25:32.842931 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:32.843007 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:32.843163 (+ 156us) server_negotiation.cc:299] Negotiation successful
0923 17:25:32.843228 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 17:25:32.844197 21904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:32.836282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:38573 (local address 127.11.229.3:35041)
0923 17:25:32.837004 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:32.837054 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:32.838418 (+ 1364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:32.838967 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:32.838974 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:32.839066 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:32.839561 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:32.839574 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:32.840364 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:32.840368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:32.841320 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:32.841330 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:32.841828 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:32.841832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:32.842000 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:32.842622 (+ 622us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:25:32.842624 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:32.842664 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:25:32.842728 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:32.843008 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:32.843026 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:32.843033 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 17:25:32.843161 (+ 128us) client_negotiation.cc:240] Negotiation successful
0923 17:25:32.843436 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":138,"threads_started":1}
I20260923 17:25:32.845296 21902 heartbeater.cc:347] Connected to a master server at 127.11.229.62:38573
I20260923 17:25:32.845502 21902 heartbeater.cc:464] Registering TS with master...
I20260923 17:25:32.845966 21902 heartbeater.cc:511] Master 127.11.229.62:38573 requested a full tablet report, sending...
I20260923 17:25:32.847002 21452 ts_manager.cc:194] Registered new tserver with Master: 0780efe01db64561a5199fa31be8dbcd (127.11.229.3:36555)
I20260923 17:25:32.847705 21452 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.229.3:35041
I20260923 17:25:32.848214 12180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 17:25:32.851081 21491 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:32.850185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37750 (local address 127.11.229.62:38573)
0923 17:25:32.850331 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:32.850336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:32.850484 (+ 148us) server_negotiation.cc:407] Connection header received
0923 17:25:32.850627 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:32.850630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:32.850674 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:32.850751 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:32.850990 (+ 239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37750: BlockingRecv error: recv got EOF from 127.0.0.1:37750 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 17:25:32.851435 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 21512
I20260923 17:25:32.857671 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 21643
I20260923 17:25:32.863515 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 21774
I20260923 17:25:32.869392 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 21421
2026-09-23T17:25:32Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (891 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-23T17:25:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:25:32Z Disabled control of system clock
I20260923 17:25:32.898739 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:41619
--webserver_interface=127.11.229.62
--webserver_port=0
--builtin_ntp_servers=127.11.229.20:46801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:41619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 17:25:33.015113 21913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:33.015398 21913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:33.015435 21913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:33.019575 21913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:25:33.019640 21913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:33.019656 21913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:33.019673 21913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:25:33.019688 21913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:25:33.024550 21913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:46801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:41619
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:41619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:25:33.025694 21913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:33.027120 21913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:33.036062 21921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:33.036059 21918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:33.036105 21913 server_base.cc:1034] running on GCE node
W20260923 17:25:33.036055 21919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:33.036970 21913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:33.038230 21913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:33.039417 21913 hybrid_clock.cc:648] HybridClock initialized: now 1790184333039384 us; error 47 us; skew 500 ppm
I20260923 17:25:33.041464 21913 webserver.cc:460] Webserver started at http://127.11.229.62:43547/ using document root <none> and password file <none>
I20260923 17:25:33.042112 21913 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:33.042212 21913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:33.042447 21913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:25:33.044553 21913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "1f84446c1d5a428e8457eeca6cc92b87"
format_stamp: "Formatted at 2026-09-23 17:25:33 on dist-test-slave-ptl9"
server_key: "c4e584cb277c7f3bd855f6609d369b38"
server_key_iv: "91a1aa9bdbb827fefa74c0b2ded41a05"
server_key_version: "encryptionkey@0"
I20260923 17:25:33.045113 21913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "1f84446c1d5a428e8457eeca6cc92b87"
format_stamp: "Formatted at 2026-09-23 17:25:33 on dist-test-slave-ptl9"
server_key: "c4e584cb277c7f3bd855f6609d369b38"
server_key_iv: "91a1aa9bdbb827fefa74c0b2ded41a05"
server_key_version: "encryptionkey@0"
I20260923 17:25:33.048903 21913 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:25:33.051450 21927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:33.052742 21913 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:25:33.052888 21913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "1f84446c1d5a428e8457eeca6cc92b87"
format_stamp: "Formatted at 2026-09-23 17:25:33 on dist-test-slave-ptl9"
server_key: "c4e584cb277c7f3bd855f6609d369b38"
server_key_iv: "91a1aa9bdbb827fefa74c0b2ded41a05"
server_key_version: "encryptionkey@0"
I20260923 17:25:33.053028 21913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:33.057922 21913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:33.058610 21913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:33.058810 21913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:33.066008 21913 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:41619
I20260923 17:25:33.066023 21979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:41619 every 8 connection(s)
I20260923 17:25:33.066999 21913 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:25:33.070497 21980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:33.074448 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 21913
I20260923 17:25:33.074573 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:25:33.074901 12180 external_mini_cluster.cc:1428] Setting key eecfaee10d565511f27fdc4ab71cb112
I20260923 17:25:33.076777 21980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f84446c1d5a428e8457eeca6cc92b87: Bootstrap starting.
I20260923 17:25:33.079236 21980 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f84446c1d5a428e8457eeca6cc92b87: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:33.080080 21980 log.cc:824] T 00000000000000000000000000000000 P 1f84446c1d5a428e8457eeca6cc92b87: Log is configured to *not* fsync() on all Append() calls
I20260923 17:25:33.080993 21983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:33.076422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37264 (local address 127.11.229.62:41619)
0923 17:25:33.077117 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:33.077134 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:33.077169 (+ 35us) server_negotiation.cc:407] Connection header received
0923 17:25:33.077920 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:33.077938 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:33.078314 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:33.078657 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:33.079488 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:33.079492 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:33.079504 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:25:33.079515 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:33.079570 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:25:33.079587 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:33.079698 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:33.079855 (+ 157us) server_negotiation.cc:299] Negotiation successful
0923 17:25:33.080095 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":143,"threads_started":1}
I20260923 17:25:33.082361 21980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f84446c1d5a428e8457eeca6cc92b87: No bootstrap required, opened a new log
I20260923 17:25:33.085986 21980 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f84446c1d5a428e8457eeca6cc92b87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f84446c1d5a428e8457eeca6cc92b87" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 41619 } }
I20260923 17:25:33.086241 21980 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f84446c1d5a428e8457eeca6cc92b87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:33.086313 21980 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f84446c1d5a428e8457eeca6cc92b87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f84446c1d5a428e8457eeca6cc92b87, State: Initialized, Role: FOLLOWER
I20260923 17:25:33.086730 21980 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f84446c1d5a428e8457eeca6cc92b87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f84446c1d5a428e8457eeca6cc92b87" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 41619 } }
I20260923 17:25:33.086877 21980 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f84446c1d5a428e8457eeca6cc92b87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:25:33.086956 21980 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f84446c1d5a428e8457eeca6cc92b87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:25:33.087071 21980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f84446c1d5a428e8457eeca6cc92b87 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:33.088238 21980 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f84446c1d5a428e8457eeca6cc92b87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f84446c1d5a428e8457eeca6cc92b87" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 41619 } }
I20260923 17:25:33.088574 21980 leader_election.cc:302] T 00000000000000000000000000000000 P 1f84446c1d5a428e8457eeca6cc92b87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f84446c1d5a428e8457eeca6cc92b87; no voters:
I20260923 17:25:33.088886 21980 leader_election.cc:288] T 00000000000000000000000000000000 P 1f84446c1d5a428e8457eeca6cc92b87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:25:33.088994 21985 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f84446c1d5a428e8457eeca6cc92b87 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:25:33.089242 21985 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f84446c1d5a428e8457eeca6cc92b87 [term 1 LEADER]: Becoming Leader. State: Replica: 1f84446c1d5a428e8457eeca6cc92b87, State: Running, Role: LEADER
I20260923 17:25:33.089612 21985 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f84446c1d5a428e8457eeca6cc92b87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f84446c1d5a428e8457eeca6cc92b87" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 41619 } }
I20260923 17:25:33.090148 21980 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f84446c1d5a428e8457eeca6cc92b87 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:25:33.091413 21987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f84446c1d5a428e8457eeca6cc92b87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f84446c1d5a428e8457eeca6cc92b87. Latest consensus state: current_term: 1 leader_uuid: "1f84446c1d5a428e8457eeca6cc92b87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f84446c1d5a428e8457eeca6cc92b87" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 41619 } } }
I20260923 17:25:33.091444 21986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f84446c1d5a428e8457eeca6cc92b87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f84446c1d5a428e8457eeca6cc92b87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f84446c1d5a428e8457eeca6cc92b87" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 41619 } } }
I20260923 17:25:33.091542 21987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f84446c1d5a428e8457eeca6cc92b87 [sys.catalog]: This master's current role is: LEADER
I20260923 17:25:33.091552 21986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f84446c1d5a428e8457eeca6cc92b87 [sys.catalog]: This master's current role is: LEADER
I20260923 17:25:33.091938 21994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:25:33.095254 21994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:25:33.100898 21994 catalog_manager.cc:1347] Generated new cluster ID: 0995644a3adb485499353929962e0063
I20260923 17:25:33.100968 21994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:25:33.112989 21994 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:25:33.113349 21994 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:25:33.120467 21994 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f84446c1d5a428e8457eeca6cc92b87: Generated new TSK 0
I20260923 17:25:33.121311 21994 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:25:33.124096 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.1:0
--local_ip_for_outbound_sockets=127.11.229.1
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:41619
--builtin_ntp_servers=127.11.229.20:46801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 17:25:33.237967 22004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:33.238234 22004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:33.238298 22004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:33.242347 22004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:33.242417 22004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:33.242499 22004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.1
I20260923 17:25:33.247264 22004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:46801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:41619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.1
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:25:33.248373 22004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:33.249861 22004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:33.260196 22010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:33.260186 22009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:33.260319 22004 server_base.cc:1034] running on GCE node
W20260923 17:25:33.260190 22012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:33.260871 22004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:33.261556 22004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:33.262722 22004 hybrid_clock.cc:648] HybridClock initialized: now 1790184333262706 us; error 28 us; skew 500 ppm
I20260923 17:25:33.264593 22004 webserver.cc:460] Webserver started at http://127.11.229.1:44141/ using document root <none> and password file <none>
I20260923 17:25:33.265187 22004 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:33.265297 22004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:33.265538 22004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:25:33.267659 22004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/data/instance:
uuid: "f5f5accbb7ee4217bee4e4be61de879e"
format_stamp: "Formatted at 2026-09-23 17:25:33 on dist-test-slave-ptl9"
server_key: "0f9b1ba718ef2272d77088b4ba92477e"
server_key_iv: "3c5c2476506d74d2071f8ad7581decc7"
server_key_version: "encryptionkey@0"
I20260923 17:25:33.268250 22004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance:
uuid: "f5f5accbb7ee4217bee4e4be61de879e"
format_stamp: "Formatted at 2026-09-23 17:25:33 on dist-test-slave-ptl9"
server_key: "0f9b1ba718ef2272d77088b4ba92477e"
server_key_iv: "3c5c2476506d74d2071f8ad7581decc7"
server_key_version: "encryptionkey@0"
I20260923 17:25:33.272065 22004 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260923 17:25:33.274636 22018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:33.275811 22004 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:25:33.275944 22004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/wal
uuid: "f5f5accbb7ee4217bee4e4be61de879e"
format_stamp: "Formatted at 2026-09-23 17:25:33 on dist-test-slave-ptl9"
server_key: "0f9b1ba718ef2272d77088b4ba92477e"
server_key_iv: "3c5c2476506d74d2071f8ad7581decc7"
server_key_version: "encryptionkey@0"
I20260923 17:25:33.276055 22004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:33.280726 22004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:33.281379 22004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:33.281555 22004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:33.282102 22004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:25:33.283025 22004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:25:33.283093 22004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:33.283162 22004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:25:33.283203 22004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:33.292090 22004 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.1:35763
I20260923 17:25:33.292115 22131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.1:35763 every 8 connection(s)
I20260923 17:25:33.293109 22004 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
I20260923 17:25:33.299398 21983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:33.295040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:57425 (local address 127.11.229.62:41619)
0923 17:25:33.295185 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:33.295189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:33.296315 (+ 1126us) server_negotiation.cc:407] Connection header received
0923 17:25:33.297762 (+ 1447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:33.297764 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:33.297805 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:33.297879 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:33.299019 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:33.299022 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:33.299023 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:25:33.299032 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:33.299048 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:25:33.299055 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:33.299125 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:33.299243 (+ 118us) server_negotiation.cc:299] Negotiation successful
0923 17:25:33.299284 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 17:25:33.300300 22134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:33.295455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41619 (local address 127.11.229.1:57425)
0923 17:25:33.296171 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:33.296212 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:33.297554 (+ 1342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:33.298082 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:33.298088 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:33.298176 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:33.298773 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:25:33.298775 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:33.298816 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:25:33.298887 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:33.299122 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:33.299128 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:33.299132 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 17:25:33.299263 (+ 131us) client_negotiation.cc:240] Negotiation successful
0923 17:25:33.299483 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":114,"threads_started":1}
I20260923 17:25:33.300724 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 22004
I20260923 17:25:33.300846 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance
I20260923 17:25:33.301195 12180 external_mini_cluster.cc:1428] Setting key 25b1318d32c50858fd5aa29e90b86d54
I20260923 17:25:33.301669 22132 heartbeater.cc:347] Connected to a master server at 127.11.229.62:41619
I20260923 17:25:33.301910 22132 heartbeater.cc:464] Registering TS with master...
I20260923 17:25:33.302336 22132 heartbeater.cc:511] Master 127.11.229.62:41619 requested a full tablet report, sending...
I20260923 17:25:33.303388 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.2:0
--local_ip_for_outbound_sockets=127.11.229.2
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:41619
--builtin_ntp_servers=127.11.229.20:46801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260923 17:25:33.304141 21944 ts_manager.cc:194] Registered new tserver with Master: f5f5accbb7ee4217bee4e4be61de879e (127.11.229.1:35763)
W20260923 17:25:33.415238 22135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:33.415504 22135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:33.415544 22135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:33.419543 22135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:33.419598 22135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:33.419654 22135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.2
I20260923 17:25:33.424332 22135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:46801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:41619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.2
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:25:33.425498 22135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:33.426906 22135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:33.436662 22141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:33.436731 22143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:33.436745 22135 server_base.cc:1034] running on GCE node
W20260923 17:25:33.436663 22140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:33.437424 22135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:33.438016 22135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:33.439162 22135 hybrid_clock.cc:648] HybridClock initialized: now 1790184333439127 us; error 48 us; skew 500 ppm
I20260923 17:25:33.440968 22135 webserver.cc:460] Webserver started at http://127.11.229.2:37883/ using document root <none> and password file <none>
I20260923 17:25:33.441632 22135 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:33.441704 22135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:33.441876 22135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:25:33.443941 22135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/data/instance:
uuid: "217795cc029844cdb2cdb7a92d07f75b"
format_stamp: "Formatted at 2026-09-23 17:25:33 on dist-test-slave-ptl9"
server_key: "b2a57f6c53539f262e6f65675a45ae6b"
server_key_iv: "65888782140de9b07a169f59316587e1"
server_key_version: "encryptionkey@0"
I20260923 17:25:33.444473 22135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance:
uuid: "217795cc029844cdb2cdb7a92d07f75b"
format_stamp: "Formatted at 2026-09-23 17:25:33 on dist-test-slave-ptl9"
server_key: "b2a57f6c53539f262e6f65675a45ae6b"
server_key_iv: "65888782140de9b07a169f59316587e1"
server_key_version: "encryptionkey@0"
I20260923 17:25:33.448750 22135 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260923 17:25:33.451454 22149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:33.452698 22135 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:25:33.452795 22135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/wal
uuid: "217795cc029844cdb2cdb7a92d07f75b"
format_stamp: "Formatted at 2026-09-23 17:25:33 on dist-test-slave-ptl9"
server_key: "b2a57f6c53539f262e6f65675a45ae6b"
server_key_iv: "65888782140de9b07a169f59316587e1"
server_key_version: "encryptionkey@0"
I20260923 17:25:33.452890 22135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:33.457598 22135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:33.458249 22135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:33.458386 22135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:33.458875 22135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:25:33.459741 22135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:25:33.459787 22135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:33.459824 22135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:25:33.459844 22135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:33.468307 22135 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.2:37301
I20260923 17:25:33.468330 22262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.2:37301 every 8 connection(s)
I20260923 17:25:33.469241 22135 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
I20260923 17:25:33.469622 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 22135
I20260923 17:25:33.469769 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance
I20260923 17:25:33.470109 12180 external_mini_cluster.cc:1428] Setting key 988f55467979b50c04454f4d706f8441
I20260923 17:25:33.472344 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.3:0
--local_ip_for_outbound_sockets=127.11.229.3
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:41619
--builtin_ntp_servers=127.11.229.20:46801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260923 17:25:33.476467 21983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:33.471368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:40335 (local address 127.11.229.62:41619)
0923 17:25:33.471519 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:33.471524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:33.472890 (+ 1366us) server_negotiation.cc:407] Connection header received
0923 17:25:33.474641 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:33.474644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:33.474681 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:33.474753 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:33.475979 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:33.475983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:33.475985 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:25:33.475999 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:33.476021 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:25:33.476034 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:33.476136 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:33.476280 (+ 144us) server_negotiation.cc:299] Negotiation successful
0923 17:25:33.476330 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 17:25:33.477401 22265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:33.471916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41619 (local address 127.11.229.2:40335)
0923 17:25:33.472748 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:33.472792 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:33.474387 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:33.474986 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:33.474997 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:33.475104 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:33.475711 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:25:33.475713 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:33.475755 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:25:33.475826 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:33.476132 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:33.476141 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:33.476149 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 17:25:33.476326 (+ 177us) client_negotiation.cc:240] Negotiation successful
0923 17:25:33.476575 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":165,"threads_started":1}
I20260923 17:25:33.478797 22263 heartbeater.cc:347] Connected to a master server at 127.11.229.62:41619
I20260923 17:25:33.479053 22263 heartbeater.cc:464] Registering TS with master...
I20260923 17:25:33.479511 22263 heartbeater.cc:511] Master 127.11.229.62:41619 requested a full tablet report, sending...
I20260923 17:25:33.480651 21944 ts_manager.cc:194] Registered new tserver with Master: 217795cc029844cdb2cdb7a92d07f75b (127.11.229.2:37301)
W20260923 17:25:33.587441 22266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:33.587738 22266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:33.587815 22266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:33.592100 22266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:33.592181 22266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:33.592269 22266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.3
I20260923 17:25:33.597299 22266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:46801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:41619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.3
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:25:33.598483 22266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:33.599922 22266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:33.609715 22271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:33.609715 22274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:33.609709 22272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:33.610116 22266 server_base.cc:1034] running on GCE node
I20260923 17:25:33.610515 22266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:33.611083 22266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:33.612267 22266 hybrid_clock.cc:648] HybridClock initialized: now 1790184333612237 us; error 48 us; skew 500 ppm
I20260923 17:25:33.614127 22266 webserver.cc:460] Webserver started at http://127.11.229.3:39997/ using document root <none> and password file <none>
I20260923 17:25:33.614732 22266 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:33.614822 22266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:33.615038 22266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:25:33.617136 22266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/data/instance:
uuid: "dd5ba011e701451d955f242378381989"
format_stamp: "Formatted at 2026-09-23 17:25:33 on dist-test-slave-ptl9"
server_key: "f5ce11ff88570c849f7d139789e3abd1"
server_key_iv: "4a5d54354816801b0820d0e5cf947199"
server_key_version: "encryptionkey@0"
I20260923 17:25:33.617883 22266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance:
uuid: "dd5ba011e701451d955f242378381989"
format_stamp: "Formatted at 2026-09-23 17:25:33 on dist-test-slave-ptl9"
server_key: "f5ce11ff88570c849f7d139789e3abd1"
server_key_iv: "4a5d54354816801b0820d0e5cf947199"
server_key_version: "encryptionkey@0"
I20260923 17:25:33.622190 22266 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:25:33.624791 22280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:33.626036 22266 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:25:33.626165 22266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/wal
uuid: "dd5ba011e701451d955f242378381989"
format_stamp: "Formatted at 2026-09-23 17:25:33 on dist-test-slave-ptl9"
server_key: "f5ce11ff88570c849f7d139789e3abd1"
server_key_iv: "4a5d54354816801b0820d0e5cf947199"
server_key_version: "encryptionkey@0"
I20260923 17:25:33.626277 22266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:33.630859 22266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:33.631464 22266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:33.631644 22266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:33.632182 22266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:25:33.633209 22266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:25:33.633294 22266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:33.633378 22266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:25:33.633423 22266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:33.642035 22266 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.3:35943
I20260923 17:25:33.642050 22393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.3:35943 every 8 connection(s)
I20260923 17:25:33.642941 22266 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
I20260923 17:25:33.648957 21983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:33.644687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:54535 (local address 127.11.229.62:41619)
0923 17:25:33.644824 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:33.644828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:33.645881 (+ 1053us) server_negotiation.cc:407] Connection header received
0923 17:25:33.647217 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:33.647221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:33.647269 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:33.647375 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:33.648484 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:33.648488 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:33.648490 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:25:33.648503 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:33.648523 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:25:33.648539 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:33.648632 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:33.648770 (+ 138us) server_negotiation.cc:299] Negotiation successful
0923 17:25:33.648818 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 17:25:33.649544 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 22266
I20260923 17:25:33.649647 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance
I20260923 17:25:33.649981 12180 external_mini_cluster.cc:1428] Setting key dfe43bd5a27d26aeb55739bda3c981fb
I20260923 17:25:33.650063 22396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:33.645052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41619 (local address 127.11.229.3:54535)
0923 17:25:33.645732 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:33.645770 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:33.646993 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:33.647546 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:33.647553 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:33.647642 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:33.648225 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:25:33.648226 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:33.648267 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:25:33.648337 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:33.648624 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:33.648633 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:33.648640 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 17:25:33.648815 (+ 175us) client_negotiation.cc:240] Negotiation successful
0923 17:25:33.649054 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":148,"threads_started":1}
I20260923 17:25:33.651270 22394 heartbeater.cc:347] Connected to a master server at 127.11.229.62:41619
I20260923 17:25:33.651508 22394 heartbeater.cc:464] Registering TS with master...
I20260923 17:25:33.651948 22394 heartbeater.cc:511] Master 127.11.229.62:41619 requested a full tablet report, sending...
I20260923 17:25:33.652953 21943 ts_manager.cc:194] Registered new tserver with Master: dd5ba011e701451d955f242378381989 (127.11.229.3:35943)
I20260923 17:25:33.663596 12180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 17:25:33.666333 21983 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:25:33.665487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37268 (local address 127.11.229.62:41619)
0923 17:25:33.665620 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:33.665625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:33.665760 (+ 135us) server_negotiation.cc:407] Connection header received
0923 17:25:33.665875 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:33.665877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:33.665908 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:33.665969 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:33.666179 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37268: BlockingRecv error: recv got EOF from 127.0.0.1:37268 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 17:25:33.666565 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 22004
I20260923 17:25:33.672640 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 22135
I20260923 17:25:33.677955 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 22266
I20260923 17:25:33.683338 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 21913
2026-09-23T17:25:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (813 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:25:33 dist-test-slave-ptl9 krb5kdc[22404](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:25:33 dist-test-slave-ptl9 krb5kdc[22404](info): set up 2 sockets
Sep 23 17:25:33 dist-test-slave-ptl9 krb5kdc[22404](info): commencing operation
krb5kdc: starting...
W20260923 17:25:34.718624 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:25:34 dist-test-slave-ptl9 krb5kdc[22404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:25:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:25:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.229.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
I20260923 17:25:34.841106 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:37709
--webserver_interface=127.11.229.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:35231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:37709
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:25:34.953840 22417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:34.954111 22417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:34.954154 22417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:34.958186 22417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:25:34.958243 22417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:34.958262 22417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:34.958278 22417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:25:34.958294 22417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:25:34.963629 22417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:35231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:37709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:37709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:25:34.964722 22417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:34.965826 22417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:34.974802 22425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:34.974795 22423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:34.975031 22417 server_base.cc:1034] running on GCE node
W20260923 17:25:34.974858 22422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:34.975711 22417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:34.976960 22417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:34.978143 22417 hybrid_clock.cc:648] HybridClock initialized: now 1790184334978106 us; error 51 us; skew 500 ppm
Sep 23 17:25:34 dist-test-slave-ptl9 krb5kdc[22404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184334, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:34.981026 22417 init.cc:377] Logged in from keytab as kudu/127.11.229.62@KRBTEST.COM (short username kudu)
I20260923 17:25:34.982023 22417 webserver.cc:460] Webserver started at http://127.11.229.62:35189/ using document root <none> and password file <none>
I20260923 17:25:34.982606 22417 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:34.982707 22417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:34.982941 22417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:25:34.985049 22417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "42585346056d4db8a419766e770b080b"
format_stamp: "Formatted at 2026-09-23 17:25:34 on dist-test-slave-ptl9"
server_key: "28fa67f88eb4edfefb46524e79cf4642"
server_key_iv: "63caa451c7be6744ac8402692e27efb7"
server_key_version: "encryptionkey@0"
I20260923 17:25:34.985670 22417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "42585346056d4db8a419766e770b080b"
format_stamp: "Formatted at 2026-09-23 17:25:34 on dist-test-slave-ptl9"
server_key: "28fa67f88eb4edfefb46524e79cf4642"
server_key_iv: "63caa451c7be6744ac8402692e27efb7"
server_key_version: "encryptionkey@0"
I20260923 17:25:34.989466 22417 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:25:34.991971 22432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:34.993208 22417 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:25:34.993367 22417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "42585346056d4db8a419766e770b080b"
format_stamp: "Formatted at 2026-09-23 17:25:34 on dist-test-slave-ptl9"
server_key: "28fa67f88eb4edfefb46524e79cf4642"
server_key_iv: "63caa451c7be6744ac8402692e27efb7"
server_key_version: "encryptionkey@0"
I20260923 17:25:34.993484 22417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:35.013083 22417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:35.016587 22417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:35.016815 22417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:35.023875 22417 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:37709
I20260923 17:25:35.024179 22484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:37709 every 8 connection(s)
I20260923 17:25:35.024907 22417 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:25:35.027803 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 22417
I20260923 17:25:35.027923 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:25:35.027997 22485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:35.028275 12180 external_mini_cluster.cc:1428] Setting key 02d04dd2a49ec7d4d16c786453e56c68
I20260923 17:25:35.034825 22485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42585346056d4db8a419766e770b080b: Bootstrap starting.
Sep 23 17:25:35 dist-test-slave-ptl9 krb5kdc[22404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:25:35.037129 22485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42585346056d4db8a419766e770b080b: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:35.038039 22485 log.cc:824] T 00000000000000000000000000000000 P 42585346056d4db8a419766e770b080b: Log is configured to *not* fsync() on all Append() calls
I20260923 17:25:35.040424 22485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42585346056d4db8a419766e770b080b: No bootstrap required, opened a new log
I20260923 17:25:35.043040 22488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:35.029817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46464 (local address 127.11.229.62:37709)
0923 17:25:35.030253 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:35.030269 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:35.030305 (+ 36us) server_negotiation.cc:407] Connection header received
0923 17:25:35.030952 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:35.030970 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:35.031306 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:35.031653 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:35.032513 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:35.033621 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:35.034495 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:35.035035 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:35.035203 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:35.037817 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:35.037827 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:35.037845 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:35.037879 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:35.039859 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:35.040278 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:35.040284 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:35.040290 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:35.040424 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:35.040707 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:35.040710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:35.040712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:35.041280 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:35.041472 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:35.041687 (+ 215us) server_negotiation.cc:299] Negotiation successful
0923 17:25:35.041996 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":136,"threads_started":1}
I20260923 17:25:35.044935 22485 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42585346056d4db8a419766e770b080b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42585346056d4db8a419766e770b080b" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 37709 } }
I20260923 17:25:35.045218 22485 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42585346056d4db8a419766e770b080b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:35.045346 22485 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42585346056d4db8a419766e770b080b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42585346056d4db8a419766e770b080b, State: Initialized, Role: FOLLOWER
I20260923 17:25:35.045789 22485 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42585346056d4db8a419766e770b080b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42585346056d4db8a419766e770b080b" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 37709 } }
I20260923 17:25:35.045933 22485 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42585346056d4db8a419766e770b080b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:25:35.045995 22485 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42585346056d4db8a419766e770b080b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:25:35.046123 22485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42585346056d4db8a419766e770b080b [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:35.047045 22485 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42585346056d4db8a419766e770b080b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42585346056d4db8a419766e770b080b" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 37709 } }
I20260923 17:25:35.047389 22485 leader_election.cc:302] T 00000000000000000000000000000000 P 42585346056d4db8a419766e770b080b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42585346056d4db8a419766e770b080b; no voters:
I20260923 17:25:35.047700 22485 leader_election.cc:288] T 00000000000000000000000000000000 P 42585346056d4db8a419766e770b080b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:25:35.047821 22490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42585346056d4db8a419766e770b080b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:25:35.048054 22490 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42585346056d4db8a419766e770b080b [term 1 LEADER]: Becoming Leader. State: Replica: 42585346056d4db8a419766e770b080b, State: Running, Role: LEADER
I20260923 17:25:35.048409 22490 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42585346056d4db8a419766e770b080b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42585346056d4db8a419766e770b080b" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 37709 } }
I20260923 17:25:35.048925 22485 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42585346056d4db8a419766e770b080b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:25:35.050109 22492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42585346056d4db8a419766e770b080b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42585346056d4db8a419766e770b080b. Latest consensus state: current_term: 1 leader_uuid: "42585346056d4db8a419766e770b080b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42585346056d4db8a419766e770b080b" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 37709 } } }
I20260923 17:25:35.050316 22492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42585346056d4db8a419766e770b080b [sys.catalog]: This master's current role is: LEADER
I20260923 17:25:35.050397 22491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42585346056d4db8a419766e770b080b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42585346056d4db8a419766e770b080b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42585346056d4db8a419766e770b080b" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 37709 } } }
I20260923 17:25:35.050484 22491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42585346056d4db8a419766e770b080b [sys.catalog]: This master's current role is: LEADER
I20260923 17:25:35.050904 22500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:25:35.054261 22500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:25:35.059507 22500 catalog_manager.cc:1347] Generated new cluster ID: eea1239a4f014455bdbe51586387ed67
I20260923 17:25:35.059583 22500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:25:35.070488 22500 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:25:35.071485 22500 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:25:35.081082 22500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42585346056d4db8a419766e770b080b: Generated new TSK 0
I20260923 17:25:35.081866 22500 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.229.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab.
Entry for principal kudu/127.11.229.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab.
I20260923 17:25:35.116879 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.1:0
--local_ip_for_outbound_sockets=127.11.229.1
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:37709
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:35231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:25:35.231016 22511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:35.231287 22511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:35.231330 22511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:35.235394 22511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:35.235455 22511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:35.235514 22511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.1
I20260923 17:25:35.240676 22511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:35231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:37709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.1
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:25:35.241745 22511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:35.242811 22511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:35.252482 22517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:35.252504 22519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:35.252504 22516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:35.252875 22511 server_base.cc:1034] running on GCE node
I20260923 17:25:35.253391 22511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:35.254029 22511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:35.255213 22511 hybrid_clock.cc:648] HybridClock initialized: now 1790184335255189 us; error 36 us; skew 500 ppm
Sep 23 17:25:35 dist-test-slave-ptl9 krb5kdc[22404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184335, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:35.257890 22511 init.cc:377] Logged in from keytab as kudu/127.11.229.1@KRBTEST.COM (short username kudu)
I20260923 17:25:35.258805 22511 webserver.cc:460] Webserver started at http://127.11.229.1:44069/ using document root <none> and password file <none>
I20260923 17:25:35.259390 22511 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:35.259477 22511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:35.259703 22511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:25:35.261849 22511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/data/instance:
uuid: "74957d0b629f4c40b96b33fe2916eb45"
format_stamp: "Formatted at 2026-09-23 17:25:35 on dist-test-slave-ptl9"
server_key: "7e70bed40fca6eef03f8ae35ffdf9a66"
server_key_iv: "5cc3e2a658e9ffe9e004da2895530eae"
server_key_version: "encryptionkey@0"
I20260923 17:25:35.262415 22511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance:
uuid: "74957d0b629f4c40b96b33fe2916eb45"
format_stamp: "Formatted at 2026-09-23 17:25:35 on dist-test-slave-ptl9"
server_key: "7e70bed40fca6eef03f8ae35ffdf9a66"
server_key_iv: "5cc3e2a658e9ffe9e004da2895530eae"
server_key_version: "encryptionkey@0"
I20260923 17:25:35.266250 22511 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:25:35.268833 22526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:35.270152 22511 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:25:35.270289 22511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/wal
uuid: "74957d0b629f4c40b96b33fe2916eb45"
format_stamp: "Formatted at 2026-09-23 17:25:35 on dist-test-slave-ptl9"
server_key: "7e70bed40fca6eef03f8ae35ffdf9a66"
server_key_iv: "5cc3e2a658e9ffe9e004da2895530eae"
server_key_version: "encryptionkey@0"
I20260923 17:25:35.270409 22511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:35.288405 22511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:35.291392 22511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:35.291611 22511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:35.292244 22511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:25:35.293147 22511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:25:35.293218 22511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:35.293313 22511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:25:35.293375 22511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:35.302364 22511 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.1:39463
I20260923 17:25:35.302387 22639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.1:39463 every 8 connection(s)
I20260923 17:25:35.303335 22511 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
I20260923 17:25:35.303872 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 22511
I20260923 17:25:35.303987 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance
I20260923 17:25:35.304288 12180 external_mini_cluster.cc:1428] Setting key 545a94fe25e044c529d2841fd5f5b04c
Sep 23 17:25:35 dist-test-slave-ptl9 krb5kdc[22404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184335, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.229.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.2@KRBTEST.COM" created.
I20260923 17:25:35.319497 22488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:35.305444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:45827 (local address 127.11.229.62:37709)
0923 17:25:35.305576 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:35.305579 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:35.306737 (+ 1158us) server_negotiation.cc:407] Connection header received
0923 17:25:35.308481 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:35.308485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:35.308539 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:35.308622 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:35.310179 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:35.310786 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:35.312125 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:35.312359 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:35.312443 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:35.315594 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:35.315597 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:35.315599 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:35.315625 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:35.317155 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:35.317676 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:35.317679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:35.317680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:35.317801 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:35.318142 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:35.318144 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:35.318145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:35.318374 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:35.318485 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:35.319066 (+ 581us) server_negotiation.cc:299] Negotiation successful
0923 17:25:35.319236 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 17:25:35.320089 22642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:35.306000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:37709 (local address 127.11.229.1:45827)
0923 17:25:35.306562 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:35.306622 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:35.308269 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:35.308820 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:35.308828 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:35.309183 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:35.309907 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:35.309930 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:35.310935 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:35.310941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:35.311998 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:35.312008 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:35.312532 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:35.312539 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:35.312797 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:35.313528 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:35.313556 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:35.315409 (+ 1853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:35.317308 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:35.317316 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:35.317333 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:35.317582 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:35.317913 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:35.317915 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:35.317917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:35.318046 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:35.318501 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:35.318511 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:35.318805 (+ 294us) client_negotiation.cc:769] Sending connection context
0923 17:25:35.318962 (+ 157us) client_negotiation.cc:240] Negotiation successful
0923 17:25:35.319224 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":117,"threads_started":1}
I20260923 17:25:35.321465 22640 heartbeater.cc:347] Connected to a master server at 127.11.229.62:37709
I20260923 17:25:35.321713 22640 heartbeater.cc:464] Registering TS with master...
I20260923 17:25:35.322225 22640 heartbeater.cc:511] Master 127.11.229.62:37709 requested a full tablet report, sending...
I20260923 17:25:35.323794 22449 ts_manager.cc:194] Registered new tserver with Master: 74957d0b629f4c40b96b33fe2916eb45 (127.11.229.1:39463)
I20260923 17:25:35.324924 22449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.1@KRBTEST.COM'} at 127.11.229.1:45827
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab.
Entry for principal kudu/127.11.229.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab.
I20260923 17:25:35.334005 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.2:0
--local_ip_for_outbound_sockets=127.11.229.2
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:37709
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:35231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:25:35.443657 22645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:35.443969 22645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:35.444047 22645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:35.448117 22645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:35.448221 22645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:35.448313 22645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.2
I20260923 17:25:35.453878 22645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:35231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:37709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.2
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:25:35.455072 22645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:35.456193 22645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:35.465878 22650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:35.465888 22651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:35.465888 22653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:35.466238 22645 server_base.cc:1034] running on GCE node
I20260923 17:25:35.466635 22645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:35.467236 22645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:35.468391 22645 hybrid_clock.cc:648] HybridClock initialized: now 1790184335468375 us; error 28 us; skew 500 ppm
Sep 23 17:25:35 dist-test-slave-ptl9 krb5kdc[22404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184335, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:35.471042 22645 init.cc:377] Logged in from keytab as kudu/127.11.229.2@KRBTEST.COM (short username kudu)
I20260923 17:25:35.471938 22645 webserver.cc:460] Webserver started at http://127.11.229.2:33327/ using document root <none> and password file <none>
I20260923 17:25:35.472500 22645 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:35.472589 22645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:35.472824 22645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:25:35.475090 22645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/data/instance:
uuid: "c901e3d3aad7443f931afc6947f9ccb2"
format_stamp: "Formatted at 2026-09-23 17:25:35 on dist-test-slave-ptl9"
server_key: "7108bb03023076e39fb44ea9caa4466f"
server_key_iv: "b34b5a126de478a134149ac7eaca2562"
server_key_version: "encryptionkey@0"
I20260923 17:25:35.475642 22645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance:
uuid: "c901e3d3aad7443f931afc6947f9ccb2"
format_stamp: "Formatted at 2026-09-23 17:25:35 on dist-test-slave-ptl9"
server_key: "7108bb03023076e39fb44ea9caa4466f"
server_key_iv: "b34b5a126de478a134149ac7eaca2562"
server_key_version: "encryptionkey@0"
I20260923 17:25:35.479398 22645 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 17:25:35.481904 22660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:35.483129 22645 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:25:35.483260 22645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/wal
uuid: "c901e3d3aad7443f931afc6947f9ccb2"
format_stamp: "Formatted at 2026-09-23 17:25:35 on dist-test-slave-ptl9"
server_key: "7108bb03023076e39fb44ea9caa4466f"
server_key_iv: "b34b5a126de478a134149ac7eaca2562"
server_key_version: "encryptionkey@0"
I20260923 17:25:35.483371 22645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:35.511209 22645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:35.514273 22645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:35.514480 22645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:35.515089 22645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:25:35.515931 22645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:25:35.515997 22645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:35.516067 22645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:25:35.516119 22645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:35.525321 22645 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.2:46725
I20260923 17:25:35.525346 22773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.2:46725 every 8 connection(s)
I20260923 17:25:35.526285 22645 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
I20260923 17:25:35.530045 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 22645
I20260923 17:25:35.530167 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance
I20260923 17:25:35.530452 12180 external_mini_cluster.cc:1428] Setting key 5b229129281a5cc9b59e6483e08e6c45
Sep 23 17:25:35 dist-test-slave-ptl9 krb5kdc[22404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184335, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.229.3@KRBTEST.COM; defaulting to no policy
I20260923 17:25:35.540561 22488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:35.528077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:35243 (local address 127.11.229.62:37709)
0923 17:25:35.528255 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:35.528260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:35.529103 (+ 843us) server_negotiation.cc:407] Connection header received
0923 17:25:35.530591 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:35.530594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:35.530654 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:35.530745 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:35.531890 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:35.532421 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:35.533620 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:35.533847 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:35.533955 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:35.536649 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:35.536660 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:35.536666 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:35.536700 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:35.538407 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:35.538927 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:35.538932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:35.538935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:35.539022 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:35.539363 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:35.539370 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:35.539372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:35.539628 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:35.539750 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:35.540203 (+ 453us) server_negotiation.cc:299] Negotiation successful
0923 17:25:35.540346 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 17:25:35.541440 22776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:35.528492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:37709 (local address 127.11.229.2:35243)
0923 17:25:35.528951 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:35.528994 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:35.530350 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:35.530914 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:35.530921 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:35.531262 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:35.531749 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:35.531763 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:35.532544 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:35.532553 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:35.533493 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:35.533504 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:35.533991 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:35.533995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:35.534158 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:35.534746 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:35.534762 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:35.536478 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:35.538539 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:35.538543 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:35.538554 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:35.538825 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:35.539129 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:35.539131 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:35.539133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:35.539268 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:35.539734 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:35.539738 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:35.539960 (+ 222us) client_negotiation.cc:769] Sending connection context
0923 17:25:35.540112 (+ 152us) client_negotiation.cc:240] Negotiation successful
0923 17:25:35.540420 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":124,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.3@KRBTEST.COM" created.
I20260923 17:25:35.542740 22774 heartbeater.cc:347] Connected to a master server at 127.11.229.62:37709
I20260923 17:25:35.542996 22774 heartbeater.cc:464] Registering TS with master...
I20260923 17:25:35.543484 22774 heartbeater.cc:511] Master 127.11.229.62:37709 requested a full tablet report, sending...
I20260923 17:25:35.544512 22449 ts_manager.cc:194] Registered new tserver with Master: c901e3d3aad7443f931afc6947f9ccb2 (127.11.229.2:46725)
I20260923 17:25:35.545082 22449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.2@KRBTEST.COM'} at 127.11.229.2:35243
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab.
Entry for principal kudu/127.11.229.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab.
I20260923 17:25:35.557651 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.3:0
--local_ip_for_outbound_sockets=127.11.229.3
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:37709
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:35231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:25:35.669891 22779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:25:35.670244 22779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:25:35.670337 22779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:25:35.674579 22779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:25:35.674669 22779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:25:35.674804 22779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.3
I20260923 17:25:35.680189 22779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:35231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:37709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.3
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:25:35.681354 22779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:25:35.682431 22779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:25:35.692011 22787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:35.692078 22785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:25:35.692082 22784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:25:35.692768 22779 server_base.cc:1034] running on GCE node
I20260923 17:25:35.693179 22779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:25:35.693832 22779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:25:35.694989 22779 hybrid_clock.cc:648] HybridClock initialized: now 1790184335694971 us; error 30 us; skew 500 ppm
Sep 23 17:25:35 dist-test-slave-ptl9 krb5kdc[22404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184335, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:25:35.697834 22779 init.cc:377] Logged in from keytab as kudu/127.11.229.3@KRBTEST.COM (short username kudu)
I20260923 17:25:35.698720 22779 webserver.cc:460] Webserver started at http://127.11.229.3:44343/ using document root <none> and password file <none>
I20260923 17:25:35.699260 22779 fs_manager.cc:359] Metadata directory not provided
I20260923 17:25:35.699318 22779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:25:35.699481 22779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:25:35.701571 22779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/data/instance:
uuid: "7825b9de7e344cc886ea3cb82f7ac839"
format_stamp: "Formatted at 2026-09-23 17:25:35 on dist-test-slave-ptl9"
server_key: "2549c838f100763e6e3cbcdd63441930"
server_key_iv: "ff8ed4ab65d2a217428148c1b3f40d0f"
server_key_version: "encryptionkey@0"
I20260923 17:25:35.702093 22779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance:
uuid: "7825b9de7e344cc886ea3cb82f7ac839"
format_stamp: "Formatted at 2026-09-23 17:25:35 on dist-test-slave-ptl9"
server_key: "2549c838f100763e6e3cbcdd63441930"
server_key_iv: "ff8ed4ab65d2a217428148c1b3f40d0f"
server_key_version: "encryptionkey@0"
I20260923 17:25:35.705742 22779 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:25:35.708084 22794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:35.709379 22779 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:25:35.709478 22779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/wal
uuid: "7825b9de7e344cc886ea3cb82f7ac839"
format_stamp: "Formatted at 2026-09-23 17:25:35 on dist-test-slave-ptl9"
server_key: "2549c838f100763e6e3cbcdd63441930"
server_key_iv: "ff8ed4ab65d2a217428148c1b3f40d0f"
server_key_version: "encryptionkey@0"
I20260923 17:25:35.709566 22779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:25:35.737708 22779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:25:35.741041 22779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:25:35.741228 22779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:25:35.741869 22779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:25:35.742662 22779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:25:35.742707 22779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:35.742748 22779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:25:35.742770 22779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:25:35.751624 22779 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.3:33359
I20260923 17:25:35.751653 22907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.3:33359 every 8 connection(s)
I20260923 17:25:35.752508 22779 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
I20260923 17:25:35.753911 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 22779
I20260923 17:25:35.754046 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance
I20260923 17:25:35.754360 12180 external_mini_cluster.cc:1428] Setting key 0f63e212db2a5c14441696f7496e331a
Sep 23 17:25:35 dist-test-slave-ptl9 krb5kdc[22404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184335, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:25:35.766031 22488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:35.754448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:35795 (local address 127.11.229.62:37709)
0923 17:25:35.754587 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:35.754592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:35.755586 (+ 994us) server_negotiation.cc:407] Connection header received
0923 17:25:35.757225 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:35.757229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:35.757310 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:35.757401 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:35.758546 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:35.759082 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:35.760216 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:35.760440 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:35.760554 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:35.762849 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:35.762858 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:35.762860 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:35.762883 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:35.764050 (+ 1167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:35.764540 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:35.764544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:35.764546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:35.764627 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:35.764959 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:35.764964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:35.764965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:35.765137 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:35.765244 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:35.765737 (+ 493us) server_negotiation.cc:299] Negotiation successful
0923 17:25:35.765860 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 17:25:35.766876 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:35.754923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:37709 (local address 127.11.229.3:35795)
0923 17:25:35.755428 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:35.755485 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:35.757009 (+ 1524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:35.757578 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:35.757585 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:35.757930 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:25:35.758408 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:35.758421 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:35.759214 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:35.759218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:35.760086 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:35.760096 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:35.760591 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:35.760595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:35.760768 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:35.761325 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:25:35.761342 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:25:35.762692 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:25:35.764174 (+ 1482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:35.764180 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:35.764194 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:35.764436 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:35.764724 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:25:35.764728 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:25:35.764730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:25:35.764865 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:25:35.765242 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:25:35.765249 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:25:35.765504 (+ 255us) client_negotiation.cc:769] Sending connection context
0923 17:25:35.765690 (+ 186us) client_negotiation.cc:240] Negotiation successful
0923 17:25:35.765997 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":117,"threads_started":1}
I20260923 17:25:35.768033 22908 heartbeater.cc:347] Connected to a master server at 127.11.229.62:37709
I20260923 17:25:35.768281 22908 heartbeater.cc:464] Registering TS with master...
I20260923 17:25:35.768796 22908 heartbeater.cc:511] Master 127.11.229.62:37709 requested a full tablet report, sending...
I20260923 17:25:35.769822 22449 ts_manager.cc:194] Registered new tserver with Master: 7825b9de7e344cc886ea3cb82f7ac839 (127.11.229.3:33359)
I20260923 17:25:35.770386 22449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.3@KRBTEST.COM'} at 127.11.229.3:35795
I20260923 17:25:35.780076 12180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:25:35.788739 22488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:35.781971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46466 (local address 127.11.229.62:37709)
0923 17:25:35.782088 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:35.782093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:35.782253 (+ 160us) server_negotiation.cc:407] Connection header received
0923 17:25:35.782353 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:35.782356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:35.782395 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:35.782462 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:25:35.783200 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:35.783704 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:35.784471 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:35.784682 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:35.784754 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:35.786081 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:25:35.786083 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:25:35.786085 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:25:35.786104 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:25:35.787367 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:35.787733 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:35.787736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:35.787737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:35.787814 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:25:35.788029 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:25:35.788031 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:25:35.788032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:25:35.788237 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:25:35.788337 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:35.788521 (+ 184us) server_negotiation.cc:299] Negotiation successful
0923 17:25:35.788616 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 17:25:35.791447 22449 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46466:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:25:35.794555 22449 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:25:35.809691 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:35.804177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34360 (local address 127.11.229.2:46725)
0923 17:25:35.804924 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:35.804938 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:35.804963 (+ 25us) server_negotiation.cc:407] Connection header received
0923 17:25:35.805039 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:35.805046 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:35.805221 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:35.805395 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:35.805593 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:35.806323 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:35.808019 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:35.808918 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:35.809036 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:35.809198 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:35.809392 (+ 194us) server_negotiation.cc:299] Negotiation successful
0923 17:25:35.809500 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":122,"threads_started":1}
I20260923 17:25:35.809677 22919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:35.803883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:46725 (local address 127.0.0.1:34360)
0923 17:25:35.804508 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:35.804522 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:35.804604 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:35.805388 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:35.805391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:35.805408 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:35.805498 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:35.805503 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:35.806462 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:35.806466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:35.807896 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:35.807907 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:35.809051 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:35.809055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:35.809173 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:35.809216 (+ 43us) client_negotiation.cc:769] Sending connection context
0923 17:25:35.809361 (+ 145us) client_negotiation.cc:240] Negotiation successful
0923 17:25:35.809497 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":87,"threads_started":1}
I20260923 17:25:35.810271 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:35.803292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47910 (local address 127.11.229.1:39463)
0923 17:25:35.803700 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:35.803716 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:35.803741 (+ 25us) server_negotiation.cc:407] Connection header received
0923 17:25:35.803994 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:35.804000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:35.804755 (+ 755us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:35.804901 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:35.805355 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:35.806507 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:35.808116 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:35.808918 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:35.809036 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:35.809198 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:35.809761 (+ 563us) server_negotiation.cc:299] Negotiation successful
0923 17:25:35.810097 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":87,"threads_started":1}
I20260923 17:25:35.810505 22920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:35.803142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:39463 (local address 127.0.0.1:47910)
0923 17:25:35.803610 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:35.803657 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:35.803781 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:35.805076 (+ 1295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:35.805079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:35.805096 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:35.805200 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:35.805206 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:35.806653 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:35.806656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:35.808003 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:35.808013 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:35.809362 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:35.809366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:35.809457 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:35.809481 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 17:25:35.809555 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 17:25:35.809626 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":46,"thread_start_us":102,"threads_started":1}
I20260923 17:25:35.811138 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:35.803275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:33359 (local address 127.0.0.1:44534)
0923 17:25:35.804170 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:35.804184 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:35.804276 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:35.804947 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:35.804953 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:35.804974 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:35.805105 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:35.805111 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:35.806699 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:35.806702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:35.808675 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:35.808684 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:35.809872 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:35.809876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:35.810829 (+ 953us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:35.810853 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 17:25:35.810934 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 17:25:35.811004 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"mutex_wait_us":22,"thread_start_us":100,"threads_started":1}
I20260923 17:25:35.811218 22923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:35.803993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44534 (local address 127.11.229.3:33359)
0923 17:25:35.804372 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:35.804406 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:35.804430 (+ 24us) server_negotiation.cc:407] Connection header received
0923 17:25:35.804504 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:35.804511 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:35.804725 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:35.804881 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:35.805288 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:35.806515 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:35.808804 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:35.809593 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:35.809720 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:35.810207 (+ 487us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:35.810969 (+ 762us) server_negotiation.cc:299] Negotiation successful
0923 17:25:35.811074 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":155,"threads_started":1}
I20260923 17:25:35.813201 22708 tablet_service.cc:1467] Processing CreateTablet for tablet c18deccb5a094eacb90584acae4a67eb (DEFAULT_TABLE table=test-table [id=2a3ad57bfe394205a894fb4b21b2d8d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:25:35.813421 22574 tablet_service.cc:1467] Processing CreateTablet for tablet c18deccb5a094eacb90584acae4a67eb (DEFAULT_TABLE table=test-table [id=2a3ad57bfe394205a894fb4b21b2d8d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:25:35.814435 22574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c18deccb5a094eacb90584acae4a67eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:35.814457 22708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c18deccb5a094eacb90584acae4a67eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:35.814265 22842 tablet_service.cc:1467] Processing CreateTablet for tablet c18deccb5a094eacb90584acae4a67eb (DEFAULT_TABLE table=test-table [id=2a3ad57bfe394205a894fb4b21b2d8d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:25:35.815231 22842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c18deccb5a094eacb90584acae4a67eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:25:35.820832 22925 tablet_bootstrap.cc:492] T c18deccb5a094eacb90584acae4a67eb P 74957d0b629f4c40b96b33fe2916eb45: Bootstrap starting.
I20260923 17:25:35.822607 22926 tablet_bootstrap.cc:492] T c18deccb5a094eacb90584acae4a67eb P c901e3d3aad7443f931afc6947f9ccb2: Bootstrap starting.
I20260923 17:25:35.823307 22925 tablet_bootstrap.cc:654] T c18deccb5a094eacb90584acae4a67eb P 74957d0b629f4c40b96b33fe2916eb45: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:35.824048 22925 log.cc:824] T c18deccb5a094eacb90584acae4a67eb P 74957d0b629f4c40b96b33fe2916eb45: Log is configured to *not* fsync() on all Append() calls
I20260923 17:25:35.824918 22926 tablet_bootstrap.cc:654] T c18deccb5a094eacb90584acae4a67eb P c901e3d3aad7443f931afc6947f9ccb2: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:35.825613 22927 tablet_bootstrap.cc:492] T c18deccb5a094eacb90584acae4a67eb P 7825b9de7e344cc886ea3cb82f7ac839: Bootstrap starting.
I20260923 17:25:35.825785 22926 log.cc:824] T c18deccb5a094eacb90584acae4a67eb P c901e3d3aad7443f931afc6947f9ccb2: Log is configured to *not* fsync() on all Append() calls
I20260923 17:25:35.826007 22925 tablet_bootstrap.cc:492] T c18deccb5a094eacb90584acae4a67eb P 74957d0b629f4c40b96b33fe2916eb45: No bootstrap required, opened a new log
I20260923 17:25:35.826234 22925 ts_tablet_manager.cc:1397] T c18deccb5a094eacb90584acae4a67eb P 74957d0b629f4c40b96b33fe2916eb45: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 17:25:35.827813 22927 tablet_bootstrap.cc:654] T c18deccb5a094eacb90584acae4a67eb P 7825b9de7e344cc886ea3cb82f7ac839: Neither blocks nor log segments found. Creating new log.
I20260923 17:25:35.827852 22926 tablet_bootstrap.cc:492] T c18deccb5a094eacb90584acae4a67eb P c901e3d3aad7443f931afc6947f9ccb2: No bootstrap required, opened a new log
I20260923 17:25:35.828027 22926 ts_tablet_manager.cc:1397] T c18deccb5a094eacb90584acae4a67eb P c901e3d3aad7443f931afc6947f9ccb2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 17:25:35.829305 22927 log.cc:824] T c18deccb5a094eacb90584acae4a67eb P 7825b9de7e344cc886ea3cb82f7ac839: Log is configured to *not* fsync() on all Append() calls
I20260923 17:25:35.830214 22925 raft_consensus.cc:348] T c18deccb5a094eacb90584acae4a67eb P 74957d0b629f4c40b96b33fe2916eb45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74957d0b629f4c40b96b33fe2916eb45" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 39463 } } peers { permanent_uuid: "7825b9de7e344cc886ea3cb82f7ac839" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33359 } } peers { permanent_uuid: "c901e3d3aad7443f931afc6947f9ccb2" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 46725 } }
I20260923 17:25:35.830464 22925 raft_consensus.cc:374] T c18deccb5a094eacb90584acae4a67eb P 74957d0b629f4c40b96b33fe2916eb45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:35.830536 22925 raft_consensus.cc:729] T c18deccb5a094eacb90584acae4a67eb P 74957d0b629f4c40b96b33fe2916eb45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74957d0b629f4c40b96b33fe2916eb45, State: Initialized, Role: FOLLOWER
I20260923 17:25:35.831004 22925 consensus_queue.cc:260] T c18deccb5a094eacb90584acae4a67eb P 74957d0b629f4c40b96b33fe2916eb45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74957d0b629f4c40b96b33fe2916eb45" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 39463 } } peers { permanent_uuid: "7825b9de7e344cc886ea3cb82f7ac839" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33359 } } peers { permanent_uuid: "c901e3d3aad7443f931afc6947f9ccb2" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 46725 } }
I20260923 17:25:35.831354 22927 tablet_bootstrap.cc:492] T c18deccb5a094eacb90584acae4a67eb P 7825b9de7e344cc886ea3cb82f7ac839: No bootstrap required, opened a new log
I20260923 17:25:35.831605 22927 ts_tablet_manager.cc:1397] T c18deccb5a094eacb90584acae4a67eb P 7825b9de7e344cc886ea3cb82f7ac839: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260923 17:25:35.831792 22640 heartbeater.cc:503] Master 127.11.229.62:37709 was elected leader, sending a full tablet report...
I20260923 17:25:35.832053 22926 raft_consensus.cc:348] T c18deccb5a094eacb90584acae4a67eb P c901e3d3aad7443f931afc6947f9ccb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74957d0b629f4c40b96b33fe2916eb45" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 39463 } } peers { permanent_uuid: "7825b9de7e344cc886ea3cb82f7ac839" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33359 } } peers { permanent_uuid: "c901e3d3aad7443f931afc6947f9ccb2" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 46725 } }
I20260923 17:25:35.832330 22926 raft_consensus.cc:374] T c18deccb5a094eacb90584acae4a67eb P c901e3d3aad7443f931afc6947f9ccb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:35.832427 22926 raft_consensus.cc:729] T c18deccb5a094eacb90584acae4a67eb P c901e3d3aad7443f931afc6947f9ccb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c901e3d3aad7443f931afc6947f9ccb2, State: Initialized, Role: FOLLOWER
I20260923 17:25:35.832876 22926 consensus_queue.cc:260] T c18deccb5a094eacb90584acae4a67eb P c901e3d3aad7443f931afc6947f9ccb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74957d0b629f4c40b96b33fe2916eb45" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 39463 } } peers { permanent_uuid: "7825b9de7e344cc886ea3cb82f7ac839" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33359 } } peers { permanent_uuid: "c901e3d3aad7443f931afc6947f9ccb2" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 46725 } }
I20260923 17:25:35.833396 22925 ts_tablet_manager.cc:1428] T c18deccb5a094eacb90584acae4a67eb P 74957d0b629f4c40b96b33fe2916eb45: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260923 17:25:35.833956 22926 ts_tablet_manager.cc:1428] T c18deccb5a094eacb90584acae4a67eb P c901e3d3aad7443f931afc6947f9ccb2: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260923 17:25:35.834034 22774 heartbeater.cc:503] Master 127.11.229.62:37709 was elected leader, sending a full tablet report...
I20260923 17:25:35.835350 22927 raft_consensus.cc:348] T c18deccb5a094eacb90584acae4a67eb P 7825b9de7e344cc886ea3cb82f7ac839 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74957d0b629f4c40b96b33fe2916eb45" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 39463 } } peers { permanent_uuid: "7825b9de7e344cc886ea3cb82f7ac839" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33359 } } peers { permanent_uuid: "c901e3d3aad7443f931afc6947f9ccb2" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 46725 } }
I20260923 17:25:35.835632 22927 raft_consensus.cc:374] T c18deccb5a094eacb90584acae4a67eb P 7825b9de7e344cc886ea3cb82f7ac839 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:25:35.835702 22927 raft_consensus.cc:729] T c18deccb5a094eacb90584acae4a67eb P 7825b9de7e344cc886ea3cb82f7ac839 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7825b9de7e344cc886ea3cb82f7ac839, State: Initialized, Role: FOLLOWER
I20260923 17:25:35.836193 22927 consensus_queue.cc:260] T c18deccb5a094eacb90584acae4a67eb P 7825b9de7e344cc886ea3cb82f7ac839 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74957d0b629f4c40b96b33fe2916eb45" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 39463 } } peers { permanent_uuid: "7825b9de7e344cc886ea3cb82f7ac839" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33359 } } peers { permanent_uuid: "c901e3d3aad7443f931afc6947f9ccb2" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 46725 } }
I20260923 17:25:35.836942 22908 heartbeater.cc:503] Master 127.11.229.62:37709 was elected leader, sending a full tablet report...
I20260923 17:25:35.837289 22927 ts_tablet_manager.cc:1428] T c18deccb5a094eacb90584acae4a67eb P 7825b9de7e344cc886ea3cb82f7ac839: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260923 17:25:35.934880 22931 raft_consensus.cc:482] T c18deccb5a094eacb90584acae4a67eb P 74957d0b629f4c40b96b33fe2916eb45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:25:35.935086 22931 raft_consensus.cc:504] T c18deccb5a094eacb90584acae4a67eb P 74957d0b629f4c40b96b33fe2916eb45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74957d0b629f4c40b96b33fe2916eb45" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 39463 } } peers { permanent_uuid: "7825b9de7e344cc886ea3cb82f7ac839" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33359 } } peers { permanent_uuid: "c901e3d3aad7443f931afc6947f9ccb2" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 46725 } }
I20260923 17:25:35.936188 22931 leader_election.cc:288] T c18deccb5a094eacb90584acae4a67eb P 74957d0b629f4c40b96b33fe2916eb45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7825b9de7e344cc886ea3cb82f7ac839 (127.11.229.3:33359), c901e3d3aad7443f931afc6947f9ccb2 (127.11.229.2:46725)
I20260923 17:25:35.940445 22923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:35.936700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:46867 (local address 127.11.229.3:33359)
0923 17:25:35.936846 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:35.936851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:35.936949 (+ 98us) server_negotiation.cc:407] Connection header received
0923 17:25:35.937136 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:35.937139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:35.937208 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:35.937326 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:35.937698 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:35.938304 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:35.939446 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:35.939851 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:35.939976 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:35.940031 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:35.940236 (+ 205us) server_negotiation.cc:299] Negotiation successful
0923 17:25:35.940313 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:25:35.940469 22935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:35.936970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:46725 (local address 127.11.229.1:33169)
0923 17:25:35.937328 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:35.937341 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:35.937433 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:35.937690 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:35.937693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:35.937711 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:35.937829 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:35.937835 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:35.938567 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:35.938569 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:35.939334 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:35.939341 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:35.939998 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:35.940001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:35.940139 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:35.940157 (+ 18us) client_negotiation.cc:769] Sending connection context
0923 17:25:35.940252 (+ 95us) client_negotiation.cc:240] Negotiation successful
0923 17:25:35.940317 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":16,"thread_start_us":89,"threads_started":1}
I20260923 17:25:35.940465 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:35.937105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:33169 (local address 127.11.229.2:46725)
0923 17:25:35.937245 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:35.937249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:35.937422 (+ 173us) server_negotiation.cc:407] Connection header received
0923 17:25:35.937521 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:35.937524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:35.937575 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:35.937658 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:25:35.937935 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:35.938458 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:35.939485 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:35.939861 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:35.939989 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:35.940040 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:35.940286 (+ 246us) server_negotiation.cc:299] Negotiation successful
0923 17:25:35.940365 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:25:35.940481 22934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:35.936452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:33359 (local address 127.11.229.1:46867)
0923 17:25:35.936807 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:35.936824 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:25:35.937014 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:25:35.937353 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:25:35.937356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:25:35.937377 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:25:35.937551 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:35.937558 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:35.938415 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:35.938418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:35.939314 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:25:35.939324 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:25:35.939985 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:25:35.939989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:25:35.940087 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:35.940133 (+ 46us) client_negotiation.cc:769] Sending connection context
0923 17:25:35.940193 (+ 60us) client_negotiation.cc:240] Negotiation successful
0923 17:25:35.940278 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":3,"thread_start_us":151,"threads_started":1}
I20260923 17:25:35.941107 22728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c18deccb5a094eacb90584acae4a67eb" candidate_uuid: "74957d0b629f4c40b96b33fe2916eb45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c901e3d3aad7443f931afc6947f9ccb2" is_pre_election: true
I20260923 17:25:35.941283 22862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c18deccb5a094eacb90584acae4a67eb" candidate_uuid: "74957d0b629f4c40b96b33fe2916eb45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7825b9de7e344cc886ea3cb82f7ac839" is_pre_election: true
I20260923 17:25:35.941458 22728 raft_consensus.cc:2454] T c18deccb5a094eacb90584acae4a67eb P c901e3d3aad7443f931afc6947f9ccb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74957d0b629f4c40b96b33fe2916eb45 in term 0.
I20260923 17:25:35.941614 22862 raft_consensus.cc:2454] T c18deccb5a094eacb90584acae4a67eb P 7825b9de7e344cc886ea3cb82f7ac839 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74957d0b629f4c40b96b33fe2916eb45 in term 0.
I20260923 17:25:35.941955 22527 leader_election.cc:302] T c18deccb5a094eacb90584acae4a67eb P 74957d0b629f4c40b96b33fe2916eb45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74957d0b629f4c40b96b33fe2916eb45, c901e3d3aad7443f931afc6947f9ccb2; no voters:
I20260923 17:25:35.942226 22931 raft_consensus.cc:2781] T c18deccb5a094eacb90584acae4a67eb P 74957d0b629f4c40b96b33fe2916eb45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:25:35.942306 22931 raft_consensus.cc:482] T c18deccb5a094eacb90584acae4a67eb P 74957d0b629f4c40b96b33fe2916eb45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:25:35.942337 22931 raft_consensus.cc:3037] T c18deccb5a094eacb90584acae4a67eb P 74957d0b629f4c40b96b33fe2916eb45 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:35.943418 22931 raft_consensus.cc:504] T c18deccb5a094eacb90584acae4a67eb P 74957d0b629f4c40b96b33fe2916eb45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74957d0b629f4c40b96b33fe2916eb45" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 39463 } } peers { permanent_uuid: "7825b9de7e344cc886ea3cb82f7ac839" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33359 } } peers { permanent_uuid: "c901e3d3aad7443f931afc6947f9ccb2" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 46725 } }
I20260923 17:25:35.943763 22931 leader_election.cc:288] T c18deccb5a094eacb90584acae4a67eb P 74957d0b629f4c40b96b33fe2916eb45 [CANDIDATE]: Term 1 election: Requested vote from peers 7825b9de7e344cc886ea3cb82f7ac839 (127.11.229.3:33359), c901e3d3aad7443f931afc6947f9ccb2 (127.11.229.2:46725)
I20260923 17:25:35.944139 22862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c18deccb5a094eacb90584acae4a67eb" candidate_uuid: "74957d0b629f4c40b96b33fe2916eb45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7825b9de7e344cc886ea3cb82f7ac839"
I20260923 17:25:35.944195 22728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c18deccb5a094eacb90584acae4a67eb" candidate_uuid: "74957d0b629f4c40b96b33fe2916eb45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c901e3d3aad7443f931afc6947f9ccb2"
I20260923 17:25:35.944247 22862 raft_consensus.cc:3037] T c18deccb5a094eacb90584acae4a67eb P 7825b9de7e344cc886ea3cb82f7ac839 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:35.944339 22728 raft_consensus.cc:3037] T c18deccb5a094eacb90584acae4a67eb P c901e3d3aad7443f931afc6947f9ccb2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:25:35.945375 22862 raft_consensus.cc:2454] T c18deccb5a094eacb90584acae4a67eb P 7825b9de7e344cc886ea3cb82f7ac839 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74957d0b629f4c40b96b33fe2916eb45 in term 1.
I20260923 17:25:35.945410 22728 raft_consensus.cc:2454] T c18deccb5a094eacb90584acae4a67eb P c901e3d3aad7443f931afc6947f9ccb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74957d0b629f4c40b96b33fe2916eb45 in term 1.
I20260923 17:25:35.945653 22527 leader_election.cc:302] T c18deccb5a094eacb90584acae4a67eb P 74957d0b629f4c40b96b33fe2916eb45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74957d0b629f4c40b96b33fe2916eb45, 7825b9de7e344cc886ea3cb82f7ac839; no voters:
I20260923 17:25:35.945840 22931 raft_consensus.cc:2781] T c18deccb5a094eacb90584acae4a67eb P 74957d0b629f4c40b96b33fe2916eb45 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:25:35.946059 22931 raft_consensus.cc:686] T c18deccb5a094eacb90584acae4a67eb P 74957d0b629f4c40b96b33fe2916eb45 [term 1 LEADER]: Becoming Leader. State: Replica: 74957d0b629f4c40b96b33fe2916eb45, State: Running, Role: LEADER
I20260923 17:25:35.946396 22931 consensus_queue.cc:237] T c18deccb5a094eacb90584acae4a67eb P 74957d0b629f4c40b96b33fe2916eb45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74957d0b629f4c40b96b33fe2916eb45" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 39463 } } peers { permanent_uuid: "7825b9de7e344cc886ea3cb82f7ac839" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33359 } } peers { permanent_uuid: "c901e3d3aad7443f931afc6947f9ccb2" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 46725 } }
I20260923 17:25:35.949771 22449 catalog_manager.cc:5495] T c18deccb5a094eacb90584acae4a67eb P 74957d0b629f4c40b96b33fe2916eb45 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74957d0b629f4c40b96b33fe2916eb45 (127.11.229.1). New cstate: current_term: 1 leader_uuid: "74957d0b629f4c40b96b33fe2916eb45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74957d0b629f4c40b96b33fe2916eb45" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 39463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7825b9de7e344cc886ea3cb82f7ac839" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c901e3d3aad7443f931afc6947f9ccb2" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 46725 } health_report { overall_health: UNKNOWN } } }
I20260923 17:25:35.964182 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:35.960678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47922 (local address 127.11.229.1:39463)
0923 17:25:35.960811 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0923 17:25:35.960816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:25:35.960829 (+ 13us) server_negotiation.cc:407] Connection header received
0923 17:25:35.960960 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:25:35.960962 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:25:35.961013 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:25:35.961113 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:25:35.961396 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:35.961896 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:35.962782 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:25:35.962999 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:25:35.963102 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:25:35.963402 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:25:35.963803 (+ 401us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:25:35.963933 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0923 17:25:35.963993 (+ 60us) server_negotiation.cc:299] Negotiation successful
0923 17:25:35.964045 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 17:25:35.969954 22862 raft_consensus.cc:1261] T c18deccb5a094eacb90584acae4a67eb P 7825b9de7e344cc886ea3cb82f7ac839 [term 1 FOLLOWER]: Refusing update from remote peer 74957d0b629f4c40b96b33fe2916eb45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:25:35.969954 22728 raft_consensus.cc:1261] T c18deccb5a094eacb90584acae4a67eb P c901e3d3aad7443f931afc6947f9ccb2 [term 1 FOLLOWER]: Refusing update from remote peer 74957d0b629f4c40b96b33fe2916eb45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:25:35.970683 22931 consensus_queue.cc:1035] T c18deccb5a094eacb90584acae4a67eb P 74957d0b629f4c40b96b33fe2916eb45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c901e3d3aad7443f931afc6947f9ccb2" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 46725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:25:35.970903 22936 consensus_queue.cc:1035] T c18deccb5a094eacb90584acae4a67eb P 74957d0b629f4c40b96b33fe2916eb45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7825b9de7e344cc886ea3cb82f7ac839" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:25:35.978075 22939 mvcc.cc:205] Tried to move back new op lower bound from 7332595040125480960 to 7332595040039391232. Current Snapshot: MvccSnapshot[applied={T|T < 7332595040125480960}]
I20260923 17:25:35.979245 22941 mvcc.cc:205] Tried to move back new op lower bound from 7332595040125480960 to 7332595040039391232. Current Snapshot: MvccSnapshot[applied={T|T < 7332595040125480960}]
I20260923 17:25:35.987555 22449 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:25:35.987771 22449 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:25:35.990326 22449 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42585346056d4db8a419766e770b080b: Sending DeleteTablet for 3 replicas of tablet c18deccb5a094eacb90584acae4a67eb
I20260923 17:25:35.991125 22574 tablet_service.cc:1514] Processing DeleteTablet for tablet c18deccb5a094eacb90584acae4a67eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:25:35 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:47910
I20260923 17:25:35.991230 22842 tablet_service.cc:1514] Processing DeleteTablet for tablet c18deccb5a094eacb90584acae4a67eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:25:35 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:44534
I20260923 17:25:35.991217 22708 tablet_service.cc:1514] Processing DeleteTablet for tablet c18deccb5a094eacb90584acae4a67eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:25:35 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:34360
I20260923 17:25:35.991684 22950 tablet_replica.cc:330] T c18deccb5a094eacb90584acae4a67eb P 74957d0b629f4c40b96b33fe2916eb45: stopping tablet replica
I20260923 17:25:35.992008 22950 raft_consensus.cc:2229] T c18deccb5a094eacb90584acae4a67eb P 74957d0b629f4c40b96b33fe2916eb45 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:25:35.992280 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 22511
I20260923 17:25:35.993132 22951 tablet_replica.cc:330] T c18deccb5a094eacb90584acae4a67eb P c901e3d3aad7443f931afc6947f9ccb2: stopping tablet replica
I20260923 17:25:35.993320 22949 tablet_replica.cc:330] T c18deccb5a094eacb90584acae4a67eb P 7825b9de7e344cc886ea3cb82f7ac839: stopping tablet replica
I20260923 17:25:35.993471 22951 raft_consensus.cc:2229] T c18deccb5a094eacb90584acae4a67eb P c901e3d3aad7443f931afc6947f9ccb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:25:35.993558 22949 raft_consensus.cc:2229] T c18deccb5a094eacb90584acae4a67eb P 7825b9de7e344cc886ea3cb82f7ac839 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:25:35.993834 22951 raft_consensus.cc:2258] T c18deccb5a094eacb90584acae4a67eb P c901e3d3aad7443f931afc6947f9ccb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:25:35.993840 22949 raft_consensus.cc:2258] T c18deccb5a094eacb90584acae4a67eb P 7825b9de7e344cc886ea3cb82f7ac839 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:25:35.995393 22949 ts_tablet_manager.cc:1905] T c18deccb5a094eacb90584acae4a67eb P 7825b9de7e344cc886ea3cb82f7ac839: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:25:35.995424 22951 ts_tablet_manager.cc:1905] T c18deccb5a094eacb90584acae4a67eb P c901e3d3aad7443f931afc6947f9ccb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:25:35.998057 22949 ts_tablet_manager.cc:1918] T c18deccb5a094eacb90584acae4a67eb P 7825b9de7e344cc886ea3cb82f7ac839: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:25:35.998183 22949 log.cc:1196] T c18deccb5a094eacb90584acae4a67eb P 7825b9de7e344cc886ea3cb82f7ac839: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-2/wal/wals/c18deccb5a094eacb90584acae4a67eb
I20260923 17:25:35.998479 22949 ts_tablet_manager.cc:1939] T c18deccb5a094eacb90584acae4a67eb P 7825b9de7e344cc886ea3cb82f7ac839: Deleting consensus metadata
I20260923 17:25:35.999105 22951 ts_tablet_manager.cc:1918] T c18deccb5a094eacb90584acae4a67eb P c901e3d3aad7443f931afc6947f9ccb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:25:35.999213 22951 log.cc:1196] T c18deccb5a094eacb90584acae4a67eb P c901e3d3aad7443f931afc6947f9ccb2: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184271597484-12180-0/minicluster-data/ts-1/wal/wals/c18deccb5a094eacb90584acae4a67eb
I20260923 17:25:35.999558 22951 ts_tablet_manager.cc:1939] T c18deccb5a094eacb90584acae4a67eb P c901e3d3aad7443f931afc6947f9ccb2: Deleting consensus metadata
I20260923 17:25:35.999716 22433 catalog_manager.cc:4841] TS 7825b9de7e344cc886ea3cb82f7ac839 (127.11.229.3:33359): tablet c18deccb5a094eacb90584acae4a67eb (table test-table [id=2a3ad57bfe394205a894fb4b21b2d8d5]) successfully deleted
I20260923 17:25:36.000803 22433 catalog_manager.cc:4841] TS c901e3d3aad7443f931afc6947f9ccb2 (127.11.229.2:46725): tablet c18deccb5a094eacb90584acae4a67eb (table test-table [id=2a3ad57bfe394205a894fb4b21b2d8d5]) successfully deleted
W20260923 17:25:36.001230 22434 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.229.1:39463 (error 108)
I20260923 17:25:36.001993 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 22645
I20260923 17:25:36.002247 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:25:36.001935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:39463 (local address 127.0.0.1:47924)
0923 17:25:36.002059 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0923 17:25:36.002136 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.1:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260923 17:25:36.002818 22434 catalog_manager.cc:4584] TS 74957d0b629f4c40b96b33fe2916eb45 (127.11.229.1:39463): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c18deccb5a094eacb90584acae4a67eb: Network error: Client connection negotiation failed: client connection to 127.11.229.1:39463: connect: Connection refused (error 111)
I20260923 17:25:36.008342 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 22779
I20260923 17:25:36.013935 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 22417
2026-09-23T17:25:36Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2331 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:25:36 dist-test-slave-ptl9 krb5kdc[22955](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:25:36 dist-test-slave-ptl9 krb5kdc[22955](info): set up 2 sockets
Sep 23 17:25:36 dist-test-slave-ptl9 krb5kdc[22955](info): commencing operation
krb5kdc: starting...
W20260923 17:25:37.050633 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:25:37 dist-test-slave-ptl9 krb5kdc[22955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:25:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:25:37Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.11.229.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.11.229.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.11.229.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/rangeradmin_127.11.229.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.11.229.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/rangeradmin_127.11.229.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.11.229.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.11.229.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.11.229.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/rangerlookup_127.11.229.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.11.229.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/rangerlookup_127.11.229.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.229.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.11.229.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.229.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/HTTP_127.11.229.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.229.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/HTTP_127.11.229.20@KRBTEST.COM.keytab.
I20260923 17:25:37.218223 12180 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task1jLtF9/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/postgres -l logfile start
2026-09-23 17:25:39.734 UTC [22985] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 17:25:39.735 UTC [22985] LOG: listening on IPv4 address "127.11.229.20", port 40117
2026-09-23 17:25:39.738 UTC [22985] LOG: listening on Unix socket "/tmp/.s.PGSQL.40117"
2026-09-23 17:25:39.761 UTC [22988] LOG: database system was shut down at 2026-09-23 17:25:38 UTC
2026-09-23 17:25:39.764 UTC [22985] LOG: database system is ready to accept connections
I20260923 17:25:40.714520 12180 mini_postgres.cc:96] Postgres bound to 40117
2026-09-23 17:25:40.721 UTC [22996] FATAL: database "slave" does not exist
127.11.229.20:40117 - accepting connections
I20260923 17:25:40.721827 12180 mini_ranger.cc:161] Starting Ranger...
I20260923 17:25:40.737160 12180 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 17:25:41.196638 12180 mini_ranger.cc:87] Created ranger Postgres database
I20260923 17:25:41.196734 12180 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ranger-admin
2026-09-23 17:25:41,668 [I] DB FLAVOR :POSTGRES
2026-09-23 17:25:41,668 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 17:25:41,668 [I] Checking connection..
2026-09-23 17:25:41,668 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:40117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 17:25:41,930 [I] Checking connection passed.
2026-09-23 17:25:41,930 [I] --------- Verifying version history table ---------
2026-09-23 17:25:41,930 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:40117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 17:25:42,167 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 17:25:42,167 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:40117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 17:25:42,423 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 17:25:42,423 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 17:25:42,423 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:40117/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 17:25:42.665 UTC [23060] WARNING: there is no transaction in progress
2026-09-23 17:25:42,671 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 17:25:42,671 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:40117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 17:25:42,925 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 17:25:42,925 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 17:25:42,925 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:40117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 17:25:43,180 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:40117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 17:25:43,443 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:40117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-ptl9.c.gcp-upstream.internal','N') ;"
2026-09-23 17:25:43,664 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:40117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 17:25:43,893 [I] Table x_portal_user does not exist in database ranger
2026-09-23 17:25:43,894 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:40117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 17:25:44,131 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 17:25:44,131 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:40117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 17:25:44,357 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:40117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 17:25:44,599 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 17:25:44,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:40117/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 17:25:45.589 UTC [23168] WARNING: there is no transaction in progress
2026-09-23 17:25:45.605 UTC [23168] WARNING: there is no transaction in progress
2026-09-23 17:25:45.622 UTC [23168] WARNING: there is no transaction in progress
2026-09-23 17:25:45.637 UTC [23168] WARNING: there is no transaction in progress
2026-09-23 17:25:45.653 UTC [23168] WARNING: there is no transaction in progress
2026-09-23 17:25:45.672 UTC [23168] WARNING: there is no transaction in progress
2026-09-23 17:25:45.737 UTC [23168] WARNING: there is no transaction in progress
2026-09-23 17:25:45.744 UTC [23168] WARNING: there is no transaction in progress
2026-09-23 17:25:45.758 UTC [23168] WARNING: there is no transaction in progress
2026-09-23 17:25:45.768 UTC [23168] WARNING: there is no transaction in progress
2026-09-23 17:25:45.778 UTC [23168] WARNING: there is no transaction in progress
2026-09-23 17:25:45.793 UTC [23168] WARNING: there is no transaction in progress
2026-09-23 17:25:45.803 UTC [23168] WARNING: there is no transaction in progress
2026-09-23 17:25:45.809 UTC [23168] WARNING: there is no transaction in progress
2026-09-23 17:25:45.816 UTC [23168] WARNING: there is no transaction in progress
2026-09-23 17:25:45.823 UTC [23168] WARNING: there is no transaction in progress
2026-09-23 17:25:46,486 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 17:25:46,487 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:40117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 17:25:46,727 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 17:25:46,728 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:40117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 17:25:46,987 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 17:25:46,987 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:40117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 17:25:47,248 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 17:25:47,248 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:40117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 17:25:47,483 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:40117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 17:25:47,761 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:40117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-ptl9.c.gcp-upstream.internal';"
2026-09-23 17:25:47,988 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 17:25:47,988 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:40117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 17:25:48,230 [I] DB_PATCHES have already been applied
I20260923 17:25:48.235689 12180 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ranger-admin:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task1jLtF9/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 17:25:48.235752 12180 mini_ranger.cc:193] Using host: 127.11.229.20
I20260923 17:25:48.239377 12180 mini_ranger.cc:227] Ranger admin URL: http://127.11.229.20:36003
Sep 23, 2026 5:25:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 5:25:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 5:25:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 5:25:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 5:25:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 5:25:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 5:25:49 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 17:25:49.359704 12180 mini_ranger.cc:160] Time spent starting Ranger: real 8.638s user 0.010s sys 0.000s
Sep 23, 2026 5:26:10 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 5:26:10 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 23, 2026 5:26:10 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 5:26:10 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 5:26:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 5:26:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 5:26:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 5:26:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 5:26:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 5:26:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 5:26:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 5:26:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 5:26:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 5:26:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 5:26:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 5:26:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 5:26:10 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 5:26:11 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 17:26:11.608493 12180 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.11.229.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.11.229.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.11.229.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/rangerkms_127.11.229.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.11.229.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/rangerkms_127.11.229.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.229.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.11.229.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.229.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/HTTP_127.11.229.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.229.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/HTTP_127.11.229.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 17:26:11 dist-test-slave-ptl9 krb5kdc[22955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184371, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 17:26:11.714511 12180 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 17:26:11.725006 12180 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 17:26:12.611326 12180 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 17:26:12.611425 12180 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ranger-kms
I20260923 17:26:12.611533 12180 mini_ranger_kms.cc:227] Using postgres at 127.11.229.20:40117
2026-09-23 17:26:13,144 [I] DB FLAVOR :POSTGRES
2026-09-23 17:26:13,144 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 17:26:13,144 [I] Checking connection
2026-09-23 17:26:13,144 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:40117/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 17:26:13,381 [I] connection success
2026-09-23 17:26:13,381 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 17:26:13,382 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 17:26:13,382 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:40117/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 17:26:13,609 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 17:26:13,609 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 17:26:13,609 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 17:26:13,609 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:40117/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 17:26:13,861 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 17:26:14.310400 12180 mini_ranger_kms.cc:310] Created kms service
I20260923 17:26:14.507964 12180 mini_ranger_kms.cc:326] Created kudu user
I20260923 17:26:14.554644 12180 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 17:26:14 dist-test-slave-ptl9 krb5kdc[22955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184371, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.11.229.20@KRBTEST.COM
I20260923 17:26:14.971823 12180 mini_ranger_kms.cc:384] Added ranger policy
I20260923 17:26:14.972000 12180 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ranger-kms:/tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 17:26:14.972038 12180 mini_ranger_kms.cc:240] Using host: 127.11.229.20
I20260923 17:26:14.975716 12180 mini_ranger_kms.cc:276] Ranger KMS PID: 23348
I20260923 17:26:14.975855 12180 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.11.229.20:38295
Sep 23, 2026 5:26:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 5:26:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 5:26:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 5:26:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 5:26:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 5:26:16 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260923 17:26:17.199638 12180 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.485s user 0.008s sys 0.214s
I20260923 17:26:17.199820 12180 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 17:26:17.199973 12180 mini_ranger_kms.cc:401] 127.11.229.20:38295/kms/v1/keys
Sep 23 17:26:17 dist-test-slave-ptl9 krb5kdc[22955](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790184377, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.229.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 23 17:26:17 dist-test-slave-ptl9 krb5kdc[22955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184377, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.229.20@KRBTEST.COM for HTTP/127.11.229.20@KRBTEST.COM
Sep 23 17:26:18 dist-test-slave-ptl9 krb5kdc[22955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184377, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.229.20@KRBTEST.COM for HTTP/127.11.229.20@KRBTEST.COM
Sep 23 17:26:20 dist-test-slave-ptl9 krb5kdc[22955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.11.229.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
I20260923 17:26:20.621134 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:36065
--webserver_interface=127.11.229.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:37301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:36065
--ranger_config_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.229.20:38295/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:26:20.740267 23399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:26:20.740586 23399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:26:20.740670 23399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:26:20.744830 23399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:26:20.744935 23399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:26:20.744973 23399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:26:20.745010 23399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:26:20.745057 23399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:26:20.750784 23399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:37301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/master-0/wal
--ranger_kms_url=127.11.229.20:38295/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:36065
--ranger_config_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:36065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:26:20.751937 23399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:26:20.753043 23399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:26:20.761760 23407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:20.761725 23404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:26:20.761906 23399 server_base.cc:1034] running on GCE node
W20260923 17:26:20.761955 23405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:26:20.762660 23399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:26:20.764024 23399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:26:20.765228 23399 hybrid_clock.cc:648] HybridClock initialized: now 1790184380765201 us; error 38 us; skew 500 ppm
Sep 23 17:26:20 dist-test-slave-ptl9 krb5kdc[22955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184380, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:26:20.768544 23399 init.cc:377] Logged in from keytab as kudu/127.11.229.62@KRBTEST.COM (short username kudu)
I20260923 17:26:20.769665 23399 webserver.cc:460] Webserver started at http://127.11.229.62:45399/ using document root <none> and password file <none>
I20260923 17:26:20.770248 23399 fs_manager.cc:359] Metadata directory not provided
I20260923 17:26:20.770305 23399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:26:20.770532 23399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:26:20 dist-test-slave-ptl9 krb5kdc[22955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184380, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for HTTP/127.11.229.20@KRBTEST.COM
I20260923 17:26:20.830672 23399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "c998afe6baf3472b84288a8df1f56bf1"
format_stamp: "Formatted at 2026-09-23 17:26:20 on dist-test-slave-ptl9"
server_key: "6d0f728def088f6c8bd9be1723f56f7a"
server_key_iv: "536645c2b226d114d3b787b190842dad"
server_key_version: "kuduclusterkey@0"
I20260923 17:26:20.831439 23399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "c998afe6baf3472b84288a8df1f56bf1"
format_stamp: "Formatted at 2026-09-23 17:26:20 on dist-test-slave-ptl9"
server_key: "6d0f728def088f6c8bd9be1723f56f7a"
server_key_iv: "536645c2b226d114d3b787b190842dad"
server_key_version: "kuduclusterkey@0"
I20260923 17:26:20.835916 23399 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 17:26:20.854260 23414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:20.856058 23399 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:26:20.856221 23399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "c998afe6baf3472b84288a8df1f56bf1"
format_stamp: "Formatted at 2026-09-23 17:26:20 on dist-test-slave-ptl9"
server_key: "6d0f728def088f6c8bd9be1723f56f7a"
server_key_iv: "536645c2b226d114d3b787b190842dad"
server_key_version: "kuduclusterkey@0"
I20260923 17:26:20.856387 23399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:26:20.883373 23399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:26:20.886149 23399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:26:20.886320 23399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:26:20.894045 23399 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:36065
I20260923 17:26:20.894060 23466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:36065 every 8 connection(s)
I20260923 17:26:20.895066 23399 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:26:20.898132 23467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:26:20.899621 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 23399
I20260923 17:26:20.899760 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:26:20.900044 12180 external_mini_cluster.cc:1428] Setting key 472558a7c522a546a1f3943d09df4550
I20260923 17:26:20.904309 23467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c998afe6baf3472b84288a8df1f56bf1: Bootstrap starting.
I20260923 17:26:20.906674 23467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c998afe6baf3472b84288a8df1f56bf1: Neither blocks nor log segments found. Creating new log.
I20260923 17:26:20.907591 23467 log.cc:824] T 00000000000000000000000000000000 P c998afe6baf3472b84288a8df1f56bf1: Log is configured to *not* fsync() on all Append() calls
Sep 23 17:26:20 dist-test-slave-ptl9 krb5kdc[22955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:26:20.909716 23467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c998afe6baf3472b84288a8df1f56bf1: No bootstrap required, opened a new log
I20260923 17:26:20.912895 23467 raft_consensus.cc:348] T 00000000000000000000000000000000 P c998afe6baf3472b84288a8df1f56bf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c998afe6baf3472b84288a8df1f56bf1" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 36065 } }
I20260923 17:26:20.913173 23467 raft_consensus.cc:374] T 00000000000000000000000000000000 P c998afe6baf3472b84288a8df1f56bf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:26:20.913252 23467 raft_consensus.cc:729] T 00000000000000000000000000000000 P c998afe6baf3472b84288a8df1f56bf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c998afe6baf3472b84288a8df1f56bf1, State: Initialized, Role: FOLLOWER
I20260923 17:26:20.913789 23467 consensus_queue.cc:260] T 00000000000000000000000000000000 P c998afe6baf3472b84288a8df1f56bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c998afe6baf3472b84288a8df1f56bf1" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 36065 } }
I20260923 17:26:20.913956 23467 raft_consensus.cc:388] T 00000000000000000000000000000000 P c998afe6baf3472b84288a8df1f56bf1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:26:20.914036 23467 raft_consensus.cc:482] T 00000000000000000000000000000000 P c998afe6baf3472b84288a8df1f56bf1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:26:20.914158 23467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c998afe6baf3472b84288a8df1f56bf1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:26:20.914484 23470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:20.901581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58378 (local address 127.11.229.62:36065)
0923 17:26:20.902013 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:20.902028 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:20.902064 (+ 36us) server_negotiation.cc:407] Connection header received
0923 17:26:20.902697 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:20.902714 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:20.903032 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:20.903373 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:20.904187 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:20.905195 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:20.906116 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:20.906595 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:20.906762 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:20.909348 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:20.909353 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:20.909367 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:26:20.909393 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:20.911554 (+ 2161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:20.912011 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:20.912017 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:20.912021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:20.912123 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:20.912424 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:20.912429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:20.912431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:20.912822 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:20.913054 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:20.913330 (+ 276us) server_negotiation.cc:299] Negotiation successful
0923 17:26:20.913632 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":138,"threads_started":1}
I20260923 17:26:20.915198 23467 raft_consensus.cc:504] T 00000000000000000000000000000000 P c998afe6baf3472b84288a8df1f56bf1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c998afe6baf3472b84288a8df1f56bf1" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 36065 } }
I20260923 17:26:20.915572 23467 leader_election.cc:302] T 00000000000000000000000000000000 P c998afe6baf3472b84288a8df1f56bf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c998afe6baf3472b84288a8df1f56bf1; no voters:
I20260923 17:26:20.915898 23467 leader_election.cc:288] T 00000000000000000000000000000000 P c998afe6baf3472b84288a8df1f56bf1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:26:20.916013 23472 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c998afe6baf3472b84288a8df1f56bf1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:26:20.916225 23472 raft_consensus.cc:686] T 00000000000000000000000000000000 P c998afe6baf3472b84288a8df1f56bf1 [term 1 LEADER]: Becoming Leader. State: Replica: c998afe6baf3472b84288a8df1f56bf1, State: Running, Role: LEADER
I20260923 17:26:20.916575 23472 consensus_queue.cc:237] T 00000000000000000000000000000000 P c998afe6baf3472b84288a8df1f56bf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c998afe6baf3472b84288a8df1f56bf1" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 36065 } }
I20260923 17:26:20.917235 23467 sys_catalog.cc:564] T 00000000000000000000000000000000 P c998afe6baf3472b84288a8df1f56bf1 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:26:20.918377 23474 sys_catalog.cc:455] T 00000000000000000000000000000000 P c998afe6baf3472b84288a8df1f56bf1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c998afe6baf3472b84288a8df1f56bf1. Latest consensus state: current_term: 1 leader_uuid: "c998afe6baf3472b84288a8df1f56bf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c998afe6baf3472b84288a8df1f56bf1" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 36065 } } }
I20260923 17:26:20.918499 23474 sys_catalog.cc:458] T 00000000000000000000000000000000 P c998afe6baf3472b84288a8df1f56bf1 [sys.catalog]: This master's current role is: LEADER
I20260923 17:26:20.918879 23473 sys_catalog.cc:455] T 00000000000000000000000000000000 P c998afe6baf3472b84288a8df1f56bf1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c998afe6baf3472b84288a8df1f56bf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c998afe6baf3472b84288a8df1f56bf1" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 36065 } } }
I20260923 17:26:20.919018 23473 sys_catalog.cc:458] T 00000000000000000000000000000000 P c998afe6baf3472b84288a8df1f56bf1 [sys.catalog]: This master's current role is: LEADER
I20260923 17:26:20.918922 23481 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:26:20.919971 23467 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 17:26:20.925177 23481 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:26:20.932238 23481 catalog_manager.cc:1347] Generated new cluster ID: 9b55f21eeef64b23bbda658fd2f9d93a
I20260923 17:26:20.932320 23481 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:26:20.949740 23481 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:26:20.951922 23481 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:26:20.957342 23481 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c998afe6baf3472b84288a8df1f56bf1: Generated new TSK 0
I20260923 17:26:20.958243 23481 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.229.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab.
Entry for principal kudu/127.11.229.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab.
I20260923 17:26:21.677959 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.229.20:38295/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.1:0
--local_ip_for_outbound_sockets=127.11.229.1
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:36065
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:37301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:26:21.801998 23512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:26:21.802264 23512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:26:21.802309 23512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:26:21.806537 23512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:26:21.806595 23512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:26:21.806655 23512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.1
I20260923 17:26:21.812192 23512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:37301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.11.229.20:38295/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:36065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.1
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:26:21.813395 23512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:26:21.814558 23512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:26:21.827281 23518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:21.827942 23519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:21.828176 23521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:26:21.828404 23512 server_base.cc:1034] running on GCE node
I20260923 17:26:21.828817 23512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:26:21.829538 23512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:26:21.830736 23512 hybrid_clock.cc:648] HybridClock initialized: now 1790184381830707 us; error 50 us; skew 500 ppm
Sep 23 17:26:21 dist-test-slave-ptl9 krb5kdc[22955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184381, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:26:21.833596 23512 init.cc:377] Logged in from keytab as kudu/127.11.229.1@KRBTEST.COM (short username kudu)
I20260923 17:26:21.834694 23512 webserver.cc:460] Webserver started at http://127.11.229.1:43503/ using document root <none> and password file <none>
I20260923 17:26:21.835444 23512 fs_manager.cc:359] Metadata directory not provided
I20260923 17:26:21.835556 23512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:26:21.835819 23512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:26:21 dist-test-slave-ptl9 krb5kdc[22955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184381, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for HTTP/127.11.229.20@KRBTEST.COM
I20260923 17:26:21.849800 23512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-0/data/instance:
uuid: "3d607a561309454cae15a5d1cece4579"
format_stamp: "Formatted at 2026-09-23 17:26:21 on dist-test-slave-ptl9"
server_key: "fdc3f5a714b2cf037fb93c7d78ad59a8"
server_key_iv: "13c70368bf21bba1f1395f02d199eea0"
server_key_version: "kuduclusterkey@0"
I20260923 17:26:21.850513 23512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance:
uuid: "3d607a561309454cae15a5d1cece4579"
format_stamp: "Formatted at 2026-09-23 17:26:21 on dist-test-slave-ptl9"
server_key: "fdc3f5a714b2cf037fb93c7d78ad59a8"
server_key_iv: "13c70368bf21bba1f1395f02d199eea0"
server_key_version: "kuduclusterkey@0"
I20260923 17:26:21.855201 23512 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:26:21.877358 23529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:21.879233 23512 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:26:21.879385 23512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-0/wal
uuid: "3d607a561309454cae15a5d1cece4579"
format_stamp: "Formatted at 2026-09-23 17:26:21 on dist-test-slave-ptl9"
server_key: "fdc3f5a714b2cf037fb93c7d78ad59a8"
server_key_iv: "13c70368bf21bba1f1395f02d199eea0"
server_key_version: "kuduclusterkey@0"
I20260923 17:26:21.879537 23512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:26:21.922089 23512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:26:21.926179 23512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:26:21.926396 23512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:26:21.927186 23512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:26:21.928246 23512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:26:21.928313 23512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:21.928395 23512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:26:21.928428 23512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:21.940250 23512 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.1:36679
I20260923 17:26:21.940272 23644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.1:36679 every 8 connection(s)
I20260923 17:26:21.941376 23512 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
I20260923 17:26:21.947706 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 23512
I20260923 17:26:21.947831 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance
I20260923 17:26:21.948175 12180 external_mini_cluster.cc:1428] Setting key d7e9df8d3e98e5295593165752877382
Sep 23 17:26:21 dist-test-slave-ptl9 krb5kdc[22955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184381, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:26:21.958261 23647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:21.943527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:38107 (local address 127.11.229.62:36065)
0923 17:26:21.943821 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:21.943830 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:21.944676 (+ 846us) server_negotiation.cc:407] Connection header received
0923 17:26:21.945797 (+ 1121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:21.945801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:21.945867 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:21.945979 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:21.947114 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:21.947897 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:21.949647 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:21.950034 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:21.950120 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:21.953450 (+ 3330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:21.953454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:21.953457 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:26:21.953484 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:21.955444 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:21.956162 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:21.956166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:21.956168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:21.956267 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:21.956683 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:21.956687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:21.956689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:21.956958 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:21.957049 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:21.957882 (+ 833us) server_negotiation.cc:299] Negotiation successful
0923 17:26:21.958054 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for kudu/127.11.229.2@KRBTEST.COM; defaulting to no policy
I20260923 17:26:21.960377 23648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:21.944039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:36065 (local address 127.11.229.1:38107)
0923 17:26:21.944517 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:21.944571 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:21.945565 (+ 994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:21.946167 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:21.946174 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:21.946451 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:26:21.946954 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:21.946968 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:21.948072 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:21.948077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:21.949496 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:21.949509 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:21.950340 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:21.950345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:21.950587 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:21.951353 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:26:21.951380 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:26:21.953215 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:26:21.955610 (+ 2395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:26:21.955617 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:26:21.955638 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:26:21.956032 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:26:21.956381 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:26:21.956384 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:26:21.956386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:26:21.956572 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:26:21.957124 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:26:21.957132 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:26:21.957535 (+ 403us) client_negotiation.cc:769] Sending connection context
0923 17:26:21.958755 (+ 1220us) client_negotiation.cc:240] Negotiation successful
0923 17:26:21.959147 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":138,"threads_started":1}
I20260923 17:26:21.961968 23645 heartbeater.cc:347] Connected to a master server at 127.11.229.62:36065
I20260923 17:26:21.962245 23645 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.2@KRBTEST.COM" created.
I20260923 17:26:21.962919 23645 heartbeater.cc:511] Master 127.11.229.62:36065 requested a full tablet report, sending...
I20260923 17:26:21.964969 23431 ts_manager.cc:194] Registered new tserver with Master: 3d607a561309454cae15a5d1cece4579 (127.11.229.1:36679)
I20260923 17:26:21.966573 23431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.1@KRBTEST.COM'} at 127.11.229.1:38107
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab.
Entry for principal kudu/127.11.229.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab.
I20260923 17:26:21.984836 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.229.20:38295/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.2:0
--local_ip_for_outbound_sockets=127.11.229.2
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:36065
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:37301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:26:22 dist-test-slave-ptl9 krb5kdc[22955](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790184382, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260923 17:26:22.124981 23651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:26:22.125365 23651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:26:22.125474 23651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:26:22.130316 23651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:26:22.130422 23651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:26:22.130541 23651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.2
I20260923 17:26:22.136428 23651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:37301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.11.229.20:38295/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:36065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.2
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:26:22.137682 23651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:26:22.138808 23651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:26:22.149094 23660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:22.149616 23657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:22.149917 23658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:26:22.150954 23651 server_base.cc:1034] running on GCE node
I20260923 17:26:22.151467 23651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:26:22.152366 23651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:26:22.153584 23651 hybrid_clock.cc:648] HybridClock initialized: now 1790184382153556 us; error 50 us; skew 500 ppm
Sep 23 17:26:22 dist-test-slave-ptl9 krb5kdc[22955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184382, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:26:22.157433 23651 init.cc:377] Logged in from keytab as kudu/127.11.229.2@KRBTEST.COM (short username kudu)
I20260923 17:26:22.158608 23651 webserver.cc:460] Webserver started at http://127.11.229.2:36503/ using document root <none> and password file <none>
I20260923 17:26:22.159351 23651 fs_manager.cc:359] Metadata directory not provided
I20260923 17:26:22.159437 23651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:26:22.159659 23651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:26:22 dist-test-slave-ptl9 krb5kdc[22955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184382, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for HTTP/127.11.229.20@KRBTEST.COM
I20260923 17:26:22.178272 23651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-1/data/instance:
uuid: "51b45ca94a0842dda7601ff144fc959d"
format_stamp: "Formatted at 2026-09-23 17:26:22 on dist-test-slave-ptl9"
server_key: "4c8c8255059af81a6a5ed753616b340d"
server_key_iv: "d1664883d4a571b68324f5df90f1b164"
server_key_version: "kuduclusterkey@0"
I20260923 17:26:22.179219 23651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance:
uuid: "51b45ca94a0842dda7601ff144fc959d"
format_stamp: "Formatted at 2026-09-23 17:26:22 on dist-test-slave-ptl9"
server_key: "4c8c8255059af81a6a5ed753616b340d"
server_key_iv: "d1664883d4a571b68324f5df90f1b164"
server_key_version: "kuduclusterkey@0"
I20260923 17:26:22.184907 23651 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 17:26:22.214720 23667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:22.216444 23651 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:26:22.216580 23651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-1/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-1/wal
uuid: "51b45ca94a0842dda7601ff144fc959d"
format_stamp: "Formatted at 2026-09-23 17:26:22 on dist-test-slave-ptl9"
server_key: "4c8c8255059af81a6a5ed753616b340d"
server_key_iv: "d1664883d4a571b68324f5df90f1b164"
server_key_version: "kuduclusterkey@0"
I20260923 17:26:22.216725 23651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:26:22.240795 23651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:26:22.244894 23651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:26:22.245173 23651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:26:22.245922 23651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:26:22.247051 23651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:26:22.247169 23651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:22.247273 23651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:26:22.247342 23651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:22.261862 23651 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.2:38427
I20260923 17:26:22.263404 23651 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
I20260923 17:26:22.264019 23780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.2:38427 every 8 connection(s)
I20260923 17:26:22.272858 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 23651
I20260923 17:26:22.273029 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance
I20260923 17:26:22.273449 12180 external_mini_cluster.cc:1428] Setting key 66a6a87f2fb0d2304074fd794b411e27
WARNING: no policy specified for kudu/127.11.229.3@KRBTEST.COM; defaulting to no policy
Sep 23 17:26:22 dist-test-slave-ptl9 krb5kdc[22955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184382, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.3@KRBTEST.COM" created.
I20260923 17:26:22.307392 23647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:22.266691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:38533 (local address 127.11.229.62:36065)
0923 17:26:22.266861 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:22.266867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:22.277911 (+ 11044us) server_negotiation.cc:407] Connection header received
0923 17:26:22.282625 (+ 4714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:22.282630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:22.282701 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:22.282801 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:22.288972 (+ 6171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:22.289872 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:22.291788 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:22.292156 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:22.292241 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:22.302619 (+ 10378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:22.302624 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:22.302628 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:26:22.302663 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:22.304545 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:22.305323 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:22.305328 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:22.305332 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:22.305440 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:22.305917 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:22.305923 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:22.305926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:22.306185 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:22.306273 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:22.307048 (+ 775us) server_negotiation.cc:299] Negotiation successful
0923 17:26:22.307195 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 17:26:22.309456 23783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:22.267242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:36065 (local address 127.11.229.2:38533)
0923 17:26:22.277702 (+ 10460us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:22.277792 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:22.279030 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:22.283210 (+ 4180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:22.283223 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:22.283664 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:26:22.284456 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:22.284482 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:22.290061 (+ 5579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:22.290065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:22.291630 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:22.291644 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:22.292490 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:22.292494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:22.292730 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:22.293601 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:26:22.293624 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:26:22.298367 (+ 4743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:26:22.304748 (+ 6381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:26:22.304755 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:26:22.304778 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:26:22.305176 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:26:22.305578 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:26:22.305583 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:26:22.305587 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:26:22.305802 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:26:22.306346 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:26:22.306353 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:26:22.306758 (+ 405us) client_negotiation.cc:769] Sending connection context
0923 17:26:22.307907 (+ 1149us) client_negotiation.cc:240] Negotiation successful
0923 17:26:22.308284 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10115,"thread_start_us":137,"threads_started":1}
I20260923 17:26:22.311070 23781 heartbeater.cc:347] Connected to a master server at 127.11.229.62:36065
I20260923 17:26:22.311374 23781 heartbeater.cc:464] Registering TS with master...
I20260923 17:26:22.311924 23781 heartbeater.cc:511] Master 127.11.229.62:36065 requested a full tablet report, sending...
I20260923 17:26:22.313251 23431 ts_manager.cc:194] Registered new tserver with Master: 51b45ca94a0842dda7601ff144fc959d (127.11.229.2:38427)
I20260923 17:26:22.314160 23431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.2@KRBTEST.COM'} at 127.11.229.2:38533
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab.
Entry for principal kudu/127.11.229.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab.
I20260923 17:26:22.333156 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.229.20:38295/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.3:0
--local_ip_for_outbound_sockets=127.11.229.3
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:36065
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:37301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:26:22 dist-test-slave-ptl9 krb5kdc[22955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184382, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for HTTP/127.11.229.20@KRBTEST.COM
W20260923 17:26:22.499846 23786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:26:22.500260 23786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:26:22.500357 23786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:26:22.506068 23786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:26:22.506215 23786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:26:22.506336 23786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.3
I20260923 17:26:22.513967 23786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:37301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.11.229.20:38295/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:36065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.3
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:26:22.515856 23786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:26:22.517411 23786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:26:22.530450 23794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:22.530453 23791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:26:22.531049 23792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:26:22.531841 23786 server_base.cc:1034] running on GCE node
I20260923 17:26:22.532359 23786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:26:22.533246 23786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:26:22.535516 23786 hybrid_clock.cc:648] HybridClock initialized: now 1790184382535443 us; error 88 us; skew 500 ppm
Sep 23 17:26:22 dist-test-slave-ptl9 krb5kdc[22955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184382, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:26:22.539266 23786 init.cc:377] Logged in from keytab as kudu/127.11.229.3@KRBTEST.COM (short username kudu)
I20260923 17:26:22.540427 23786 webserver.cc:460] Webserver started at http://127.11.229.3:39435/ using document root <none> and password file <none>
I20260923 17:26:22.541245 23786 fs_manager.cc:359] Metadata directory not provided
I20260923 17:26:22.541412 23786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:26:22.541703 23786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:26:22 dist-test-slave-ptl9 krb5kdc[22955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184382, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for HTTP/127.11.229.20@KRBTEST.COM
I20260923 17:26:22.558842 23786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-2/data/instance:
uuid: "ffc9beaa8dfc436492ec2ffef17bb6da"
format_stamp: "Formatted at 2026-09-23 17:26:22 on dist-test-slave-ptl9"
server_key: "fad2ba05cde128fab0bfea5f06d671b4"
server_key_iv: "6037de3370021f7a028a6817496b3f39"
server_key_version: "kuduclusterkey@0"
I20260923 17:26:22.559733 23786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance:
uuid: "ffc9beaa8dfc436492ec2ffef17bb6da"
format_stamp: "Formatted at 2026-09-23 17:26:22 on dist-test-slave-ptl9"
server_key: "fad2ba05cde128fab0bfea5f06d671b4"
server_key_iv: "6037de3370021f7a028a6817496b3f39"
server_key_version: "kuduclusterkey@0"
I20260923 17:26:22.570485 23786 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.003s sys 0.001s
I20260923 17:26:22.588243 23801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:22.590283 23786 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260923 17:26:22.590433 23786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-2/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-2/wal
uuid: "ffc9beaa8dfc436492ec2ffef17bb6da"
format_stamp: "Formatted at 2026-09-23 17:26:22 on dist-test-slave-ptl9"
server_key: "fad2ba05cde128fab0bfea5f06d671b4"
server_key_iv: "6037de3370021f7a028a6817496b3f39"
server_key_version: "kuduclusterkey@0"
I20260923 17:26:22.590581 23786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:26:22.629408 23786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:26:22.633298 23786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:26:22.633550 23786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:26:22.634312 23786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:26:22.635365 23786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:26:22.635459 23786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:22.635550 23786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:26:22.635608 23786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:26:22.647014 23786 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.3:43177
I20260923 17:26:22.648576 23786 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
I20260923 17:26:22.652734 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 23786
I20260923 17:26:22.652869 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance
I20260923 17:26:22.653242 12180 external_mini_cluster.cc:1428] Setting key d0f8902fe7cb02d09a95c0752cfc5b9e
I20260923 17:26:22.653465 23914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.3:43177 every 8 connection(s)
Sep 23 17:26:22 dist-test-slave-ptl9 krb5kdc[22955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184382, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:26:22.668195 23647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:22.652737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:33469 (local address 127.11.229.62:36065)
0923 17:26:22.652865 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:22.652870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:22.653528 (+ 658us) server_negotiation.cc:407] Connection header received
0923 17:26:22.655050 (+ 1522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:22.655053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:22.655106 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:22.655153 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:22.656993 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:22.657832 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:22.659558 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:22.659880 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:22.659948 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:22.663590 (+ 3642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:22.663594 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:22.663597 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:26:22.663623 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:22.665332 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:22.666054 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:22.666059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:22.666061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:22.666154 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:22.666817 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:22.666835 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:22.666837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:22.667076 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:22.667156 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:22.667907 (+ 751us) server_negotiation.cc:299] Negotiation successful
0923 17:26:22.668034 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 17:26:22.669350 23917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:22.651512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:36065 (local address 127.11.229.3:33469)
0923 17:26:22.653247 (+ 1735us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:22.653411 (+ 164us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:22.654792 (+ 1381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:22.655571 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:22.655580 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:22.656057 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:26:22.656830 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:22.656850 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:22.657983 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:22.657986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:22.659421 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:22.659432 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:22.660161 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:22.660165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:22.660398 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:22.661304 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:26:22.661326 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:26:22.663409 (+ 2083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:26:22.665457 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:26:22.665466 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:26:22.665485 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:26:22.665904 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:26:22.666467 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:26:22.666470 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:26:22.666472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:26:22.666692 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:26:22.667245 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:26:22.667252 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:26:22.667595 (+ 343us) client_negotiation.cc:769] Sending connection context
0923 17:26:22.667849 (+ 254us) client_negotiation.cc:240] Negotiation successful
0923 17:26:22.668187 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1477,"thread_start_us":221,"threads_started":1}
I20260923 17:26:22.670871 23915 heartbeater.cc:347] Connected to a master server at 127.11.229.62:36065
I20260923 17:26:22.671159 23915 heartbeater.cc:464] Registering TS with master...
I20260923 17:26:22.671769 23915 heartbeater.cc:511] Master 127.11.229.62:36065 requested a full tablet report, sending...
I20260923 17:26:22.673061 23431 ts_manager.cc:194] Registered new tserver with Master: ffc9beaa8dfc436492ec2ffef17bb6da (127.11.229.3:43177)
I20260923 17:26:22.673841 23431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.3@KRBTEST.COM'} at 127.11.229.3:33469
I20260923 17:26:22.679494 12180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:26:22.697293 23647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:22.682036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58390 (local address 127.11.229.62:36065)
0923 17:26:22.682156 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:22.682160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:22.683666 (+ 1506us) server_negotiation.cc:407] Connection header received
0923 17:26:22.683819 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:22.683822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:22.683871 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:22.683953 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:26:22.685142 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:22.685970 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:22.687260 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:22.687577 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:22.687688 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:22.693323 (+ 5635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:26:22.693327 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:26:22.693330 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:26:22.693359 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:26:22.695191 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:22.695713 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:22.695717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:22.695719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:22.695813 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:26:22.696145 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:26:22.696149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:26:22.696151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:26:22.696391 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:26:22.696571 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:22.696857 (+ 286us) server_negotiation.cc:299] Negotiation successful
0923 17:26:22.697092 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 17:26:22.701319 23431 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:26:22.706014 23431 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:26:22.730656 23930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:22.720025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:38427 (local address 127.0.0.1:48138)
0923 17:26:22.722610 (+ 2585us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:22.722734 (+ 124us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:22.722875 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:22.723615 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:22.723621 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:22.723646 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:26:22.723801 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:22.723807 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:22.726212 (+ 2405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:22.726215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:22.727563 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:22.727575 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:22.730099 (+ 2524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:22.730103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:22.730215 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:22.730262 (+ 47us) client_negotiation.cc:769] Sending connection context
0923 17:26:22.730358 (+ 96us) client_negotiation.cc:240] Negotiation successful
0923 17:26:22.730482 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2383,"mutex_wait_us":20,"spinlock_wait_cycles":3456,"thread_start_us":159,"threads_started":1}
I20260923 17:26:22.731796 23928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:22.720143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48138 (local address 127.11.229.2:38427)
0923 17:26:22.723031 (+ 2888us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:22.723078 (+ 47us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:22.723103 (+ 25us) server_negotiation.cc:407] Connection header received
0923 17:26:22.723184 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:22.723190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:22.723409 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:22.723959 (+ 550us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:26:22.724100 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:22.726088 (+ 1988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:22.727785 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:22.728720 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:22.731330 (+ 2610us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:22.731492 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:22.731548 (+ 56us) server_negotiation.cc:299] Negotiation successful
0923 17:26:22.731650 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2777,"thread_start_us":156,"threads_started":1}
I20260923 17:26:22.732870 23929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:22.721496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51802 (local address 127.11.229.1:36679)
0923 17:26:22.723265 (+ 1769us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:22.723272 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:22.726648 (+ 3376us) server_negotiation.cc:407] Connection header received
0923 17:26:22.726850 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:22.726856 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:22.727048 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:22.727155 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:26:22.727578 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:22.728746 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:22.730965 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:22.731872 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:22.731968 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:22.732148 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:22.732607 (+ 459us) server_negotiation.cc:299] Negotiation successful
0923 17:26:22.732704 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1659,"thread_start_us":128,"threads_started":1}
I20260923 17:26:22.733322 23927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:22.721841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:36679 (local address 127.0.0.1:51802)
0923 17:26:22.726520 (+ 4679us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:22.726534 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:22.726718 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:22.727323 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:22.727326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:22.727342 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:26:22.727456 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:22.727461 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:22.728880 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:22.728883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:22.730826 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:22.730837 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:22.732269 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:22.732273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:22.732429 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:22.732461 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 17:26:22.733098 (+ 637us) client_negotiation.cc:240] Negotiation successful
0923 17:26:22.733178 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4619,"mutex_wait_us":1,"thread_start_us":154,"threads_started":1}
I20260923 17:26:22.735405 23714 tablet_service.cc:1467] Processing CreateTablet for tablet 50ffa350c7fa44fbae946515f401ab9d (DEFAULT_TABLE table=test-table [id=8424451c9d1a46e4b42ad5fac789c7ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:26:22.736320 23561 tablet_service.cc:1467] Processing CreateTablet for tablet 50ffa350c7fa44fbae946515f401ab9d (DEFAULT_TABLE table=test-table [id=8424451c9d1a46e4b42ad5fac789c7ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:26:22.736951 23714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50ffa350c7fa44fbae946515f401ab9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:26:22.737483 23561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50ffa350c7fa44fbae946515f401ab9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:26:22.746610 23932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:22.721626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:43177 (local address 127.0.0.1:52416)
0923 17:26:22.723470 (+ 1844us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:22.723484 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:22.723677 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:22.724382 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:22.724385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:22.724403 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:26:22.724484 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:22.724490 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:22.726037 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:22.726040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:22.733189 (+ 7149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:22.733200 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:22.746091 (+ 12891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:22.746094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:22.746208 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:22.746236 (+ 28us) client_negotiation.cc:769] Sending connection context
0923 17:26:22.746323 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 17:26:22.746426 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1778,"mutex_wait_us":3146,"thread_start_us":131,"threads_started":1}
I20260923 17:26:22.746688 23931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:22.721794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52416 (local address 127.11.229.3:43177)
0923 17:26:22.722287 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:22.722324 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:22.723597 (+ 1273us) server_negotiation.cc:407] Connection header received
0923 17:26:22.723823 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:22.723829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:22.724030 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:22.724202 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:26:22.724614 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:22.725896 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:22.741368 (+ 15472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:22.742340 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:22.742448 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:22.742602 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:22.746388 (+ 3786us) server_negotiation.cc:299] Negotiation successful
0923 17:26:22.746517 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":108,"threads_started":1}
I20260923 17:26:22.748157 23934 tablet_bootstrap.cc:492] T 50ffa350c7fa44fbae946515f401ab9d P 3d607a561309454cae15a5d1cece4579: Bootstrap starting.
I20260923 17:26:22.750674 23934 tablet_bootstrap.cc:654] T 50ffa350c7fa44fbae946515f401ab9d P 3d607a561309454cae15a5d1cece4579: Neither blocks nor log segments found. Creating new log.
I20260923 17:26:22.750213 23849 tablet_service.cc:1467] Processing CreateTablet for tablet 50ffa350c7fa44fbae946515f401ab9d (DEFAULT_TABLE table=test-table [id=8424451c9d1a46e4b42ad5fac789c7ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:26:22.751617 23934 log.cc:824] T 50ffa350c7fa44fbae946515f401ab9d P 3d607a561309454cae15a5d1cece4579: Log is configured to *not* fsync() on all Append() calls
I20260923 17:26:22.751768 23849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50ffa350c7fa44fbae946515f401ab9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:26:22.752415 23933 tablet_bootstrap.cc:492] T 50ffa350c7fa44fbae946515f401ab9d P 51b45ca94a0842dda7601ff144fc959d: Bootstrap starting.
I20260923 17:26:22.755985 23933 tablet_bootstrap.cc:654] T 50ffa350c7fa44fbae946515f401ab9d P 51b45ca94a0842dda7601ff144fc959d: Neither blocks nor log segments found. Creating new log.
I20260923 17:26:22.756930 23933 log.cc:824] T 50ffa350c7fa44fbae946515f401ab9d P 51b45ca94a0842dda7601ff144fc959d: Log is configured to *not* fsync() on all Append() calls
I20260923 17:26:22.759387 23934 tablet_bootstrap.cc:492] T 50ffa350c7fa44fbae946515f401ab9d P 3d607a561309454cae15a5d1cece4579: No bootstrap required, opened a new log
I20260923 17:26:22.759614 23934 ts_tablet_manager.cc:1397] T 50ffa350c7fa44fbae946515f401ab9d P 3d607a561309454cae15a5d1cece4579: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260923 17:26:22.761138 23937 tablet_bootstrap.cc:492] T 50ffa350c7fa44fbae946515f401ab9d P ffc9beaa8dfc436492ec2ffef17bb6da: Bootstrap starting.
I20260923 17:26:22.764405 23937 tablet_bootstrap.cc:654] T 50ffa350c7fa44fbae946515f401ab9d P ffc9beaa8dfc436492ec2ffef17bb6da: Neither blocks nor log segments found. Creating new log.
I20260923 17:26:22.764271 23934 raft_consensus.cc:348] T 50ffa350c7fa44fbae946515f401ab9d P 3d607a561309454cae15a5d1cece4579 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b45ca94a0842dda7601ff144fc959d" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 38427 } } peers { permanent_uuid: "3d607a561309454cae15a5d1cece4579" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36679 } } peers { permanent_uuid: "ffc9beaa8dfc436492ec2ffef17bb6da" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 43177 } }
I20260923 17:26:22.764567 23934 raft_consensus.cc:374] T 50ffa350c7fa44fbae946515f401ab9d P 3d607a561309454cae15a5d1cece4579 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:26:22.764616 23934 raft_consensus.cc:729] T 50ffa350c7fa44fbae946515f401ab9d P 3d607a561309454cae15a5d1cece4579 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d607a561309454cae15a5d1cece4579, State: Initialized, Role: FOLLOWER
I20260923 17:26:22.765323 23937 log.cc:824] T 50ffa350c7fa44fbae946515f401ab9d P ffc9beaa8dfc436492ec2ffef17bb6da: Log is configured to *not* fsync() on all Append() calls
I20260923 17:26:22.766547 23934 consensus_queue.cc:260] T 50ffa350c7fa44fbae946515f401ab9d P 3d607a561309454cae15a5d1cece4579 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b45ca94a0842dda7601ff144fc959d" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 38427 } } peers { permanent_uuid: "3d607a561309454cae15a5d1cece4579" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36679 } } peers { permanent_uuid: "ffc9beaa8dfc436492ec2ffef17bb6da" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 43177 } }
I20260923 17:26:22.767759 23937 tablet_bootstrap.cc:492] T 50ffa350c7fa44fbae946515f401ab9d P ffc9beaa8dfc436492ec2ffef17bb6da: No bootstrap required, opened a new log
I20260923 17:26:22.767791 23934 ts_tablet_manager.cc:1428] T 50ffa350c7fa44fbae946515f401ab9d P 3d607a561309454cae15a5d1cece4579: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260923 17:26:22.767966 23937 ts_tablet_manager.cc:1397] T 50ffa350c7fa44fbae946515f401ab9d P ffc9beaa8dfc436492ec2ffef17bb6da: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260923 17:26:22.768183 23645 heartbeater.cc:503] Master 127.11.229.62:36065 was elected leader, sending a full tablet report...
I20260923 17:26:22.770016 23933 tablet_bootstrap.cc:492] T 50ffa350c7fa44fbae946515f401ab9d P 51b45ca94a0842dda7601ff144fc959d: No bootstrap required, opened a new log
I20260923 17:26:22.770222 23933 ts_tablet_manager.cc:1397] T 50ffa350c7fa44fbae946515f401ab9d P 51b45ca94a0842dda7601ff144fc959d: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.004s
I20260923 17:26:22.772490 23937 raft_consensus.cc:348] T 50ffa350c7fa44fbae946515f401ab9d P ffc9beaa8dfc436492ec2ffef17bb6da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b45ca94a0842dda7601ff144fc959d" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 38427 } } peers { permanent_uuid: "3d607a561309454cae15a5d1cece4579" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36679 } } peers { permanent_uuid: "ffc9beaa8dfc436492ec2ffef17bb6da" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 43177 } }
I20260923 17:26:22.772770 23937 raft_consensus.cc:374] T 50ffa350c7fa44fbae946515f401ab9d P ffc9beaa8dfc436492ec2ffef17bb6da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:26:22.772823 23937 raft_consensus.cc:729] T 50ffa350c7fa44fbae946515f401ab9d P ffc9beaa8dfc436492ec2ffef17bb6da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffc9beaa8dfc436492ec2ffef17bb6da, State: Initialized, Role: FOLLOWER
I20260923 17:26:22.773399 23937 consensus_queue.cc:260] T 50ffa350c7fa44fbae946515f401ab9d P ffc9beaa8dfc436492ec2ffef17bb6da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b45ca94a0842dda7601ff144fc959d" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 38427 } } peers { permanent_uuid: "3d607a561309454cae15a5d1cece4579" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36679 } } peers { permanent_uuid: "ffc9beaa8dfc436492ec2ffef17bb6da" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 43177 } }
I20260923 17:26:22.774515 23937 ts_tablet_manager.cc:1428] T 50ffa350c7fa44fbae946515f401ab9d P ffc9beaa8dfc436492ec2ffef17bb6da: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260923 17:26:22.774878 23915 heartbeater.cc:503] Master 127.11.229.62:36065 was elected leader, sending a full tablet report...
I20260923 17:26:22.776909 23933 raft_consensus.cc:348] T 50ffa350c7fa44fbae946515f401ab9d P 51b45ca94a0842dda7601ff144fc959d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b45ca94a0842dda7601ff144fc959d" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 38427 } } peers { permanent_uuid: "3d607a561309454cae15a5d1cece4579" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36679 } } peers { permanent_uuid: "ffc9beaa8dfc436492ec2ffef17bb6da" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 43177 } }
I20260923 17:26:22.777309 23933 raft_consensus.cc:374] T 50ffa350c7fa44fbae946515f401ab9d P 51b45ca94a0842dda7601ff144fc959d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:26:22.777408 23933 raft_consensus.cc:729] T 50ffa350c7fa44fbae946515f401ab9d P 51b45ca94a0842dda7601ff144fc959d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b45ca94a0842dda7601ff144fc959d, State: Initialized, Role: FOLLOWER
I20260923 17:26:22.779894 23933 consensus_queue.cc:260] T 50ffa350c7fa44fbae946515f401ab9d P 51b45ca94a0842dda7601ff144fc959d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b45ca94a0842dda7601ff144fc959d" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 38427 } } peers { permanent_uuid: "3d607a561309454cae15a5d1cece4579" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36679 } } peers { permanent_uuid: "ffc9beaa8dfc436492ec2ffef17bb6da" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 43177 } }
I20260923 17:26:22.781215 23933 ts_tablet_manager.cc:1428] T 50ffa350c7fa44fbae946515f401ab9d P 51b45ca94a0842dda7601ff144fc959d: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20260923 17:26:22.781708 23781 heartbeater.cc:503] Master 127.11.229.62:36065 was elected leader, sending a full tablet report...
Sep 23 17:26:22 dist-test-slave-ptl9 krb5kdc[22955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184382, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for HTTP/127.11.229.20@KRBTEST.COM
I20260923 17:26:22.869793 23941 raft_consensus.cc:482] T 50ffa350c7fa44fbae946515f401ab9d P 51b45ca94a0842dda7601ff144fc959d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:26:22.870041 23941 raft_consensus.cc:504] T 50ffa350c7fa44fbae946515f401ab9d P 51b45ca94a0842dda7601ff144fc959d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b45ca94a0842dda7601ff144fc959d" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 38427 } } peers { permanent_uuid: "3d607a561309454cae15a5d1cece4579" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36679 } } peers { permanent_uuid: "ffc9beaa8dfc436492ec2ffef17bb6da" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 43177 } }
I20260923 17:26:22.871536 23941 leader_election.cc:288] T 50ffa350c7fa44fbae946515f401ab9d P 51b45ca94a0842dda7601ff144fc959d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d607a561309454cae15a5d1cece4579 (127.11.229.1:36679), ffc9beaa8dfc436492ec2ffef17bb6da (127.11.229.3:43177)
I20260923 17:26:22.877624 23929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:22.872945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:42653 (local address 127.11.229.1:36679)
0923 17:26:22.873104 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:22.873108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:22.873120 (+ 12us) server_negotiation.cc:407] Connection header received
0923 17:26:22.873173 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:22.873175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:22.873211 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:22.873321 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:26:22.873601 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:22.874402 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:22.876184 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:22.876820 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:22.876920 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:22.876980 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:22.877395 (+ 415us) server_negotiation.cc:299] Negotiation successful
0923 17:26:22.877483 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 17:26:22.878237 23945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:22.872481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:36679 (local address 127.11.229.2:42653)
0923 17:26:22.872869 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:22.872883 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:22.872976 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:22.873334 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:22.873337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:22.873354 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:26:22.873473 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:22.873479 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:22.874558 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:22.874561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:22.876000 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:22.876013 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:22.877103 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:22.877106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:22.877204 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:22.877229 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 17:26:22.877979 (+ 750us) client_negotiation.cc:240] Negotiation successful
0923 17:26:22.878064 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":184,"threads_started":1}
I20260923 17:26:22.879084 23599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ffa350c7fa44fbae946515f401ab9d" candidate_uuid: "51b45ca94a0842dda7601ff144fc959d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d607a561309454cae15a5d1cece4579" is_pre_election: true
I20260923 17:26:22.879452 23931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:22.872381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:49403 (local address 127.11.229.3:43177)
0923 17:26:22.872508 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:22.872515 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:22.872530 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:26:22.872597 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:22.872600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:22.872650 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:22.872734 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:26:22.875801 (+ 3067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:22.876639 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:22.878147 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:22.878753 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:22.878856 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:22.878903 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:22.879251 (+ 348us) server_negotiation.cc:299] Negotiation successful
0923 17:26:22.879329 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 17:26:22.879442 23599 raft_consensus.cc:2454] T 50ffa350c7fa44fbae946515f401ab9d P 3d607a561309454cae15a5d1cece4579 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51b45ca94a0842dda7601ff144fc959d in term 0.
I20260923 17:26:22.879869 23944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:22.871944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:43177 (local address 127.11.229.2:49403)
0923 17:26:22.872242 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:22.872258 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:22.872382 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:22.875472 (+ 3090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:26:22.875476 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:26:22.875498 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:26:22.875628 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:22.875635 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:22.876816 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:22.876819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:22.877988 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:26:22.878000 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:26:22.879013 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:26:22.879016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:26:22.879119 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:22.879138 (+ 19us) client_negotiation.cc:769] Sending connection context
0923 17:26:22.879656 (+ 518us) client_negotiation.cc:240] Negotiation successful
0923 17:26:22.879734 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":128,"threads_started":1}
I20260923 17:26:22.880084 23671 leader_election.cc:302] T 50ffa350c7fa44fbae946515f401ab9d P 51b45ca94a0842dda7601ff144fc959d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d607a561309454cae15a5d1cece4579, 51b45ca94a0842dda7601ff144fc959d; no voters:
I20260923 17:26:22.880424 23941 raft_consensus.cc:2781] T 50ffa350c7fa44fbae946515f401ab9d P 51b45ca94a0842dda7601ff144fc959d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:26:22.880514 23941 raft_consensus.cc:482] T 50ffa350c7fa44fbae946515f401ab9d P 51b45ca94a0842dda7601ff144fc959d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:26:22.880563 23941 raft_consensus.cc:3037] T 50ffa350c7fa44fbae946515f401ab9d P 51b45ca94a0842dda7601ff144fc959d [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:26:22.880637 23866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ffa350c7fa44fbae946515f401ab9d" candidate_uuid: "51b45ca94a0842dda7601ff144fc959d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffc9beaa8dfc436492ec2ffef17bb6da" is_pre_election: true
I20260923 17:26:22.880935 23866 raft_consensus.cc:2454] T 50ffa350c7fa44fbae946515f401ab9d P ffc9beaa8dfc436492ec2ffef17bb6da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51b45ca94a0842dda7601ff144fc959d in term 0.
I20260923 17:26:22.881932 23941 raft_consensus.cc:504] T 50ffa350c7fa44fbae946515f401ab9d P 51b45ca94a0842dda7601ff144fc959d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b45ca94a0842dda7601ff144fc959d" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 38427 } } peers { permanent_uuid: "3d607a561309454cae15a5d1cece4579" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36679 } } peers { permanent_uuid: "ffc9beaa8dfc436492ec2ffef17bb6da" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 43177 } }
I20260923 17:26:22.882339 23941 leader_election.cc:288] T 50ffa350c7fa44fbae946515f401ab9d P 51b45ca94a0842dda7601ff144fc959d [CANDIDATE]: Term 1 election: Requested vote from peers 3d607a561309454cae15a5d1cece4579 (127.11.229.1:36679), ffc9beaa8dfc436492ec2ffef17bb6da (127.11.229.3:43177)
I20260923 17:26:22.882826 23599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ffa350c7fa44fbae946515f401ab9d" candidate_uuid: "51b45ca94a0842dda7601ff144fc959d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d607a561309454cae15a5d1cece4579"
I20260923 17:26:22.882889 23866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ffa350c7fa44fbae946515f401ab9d" candidate_uuid: "51b45ca94a0842dda7601ff144fc959d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffc9beaa8dfc436492ec2ffef17bb6da"
I20260923 17:26:22.882953 23599 raft_consensus.cc:3037] T 50ffa350c7fa44fbae946515f401ab9d P 3d607a561309454cae15a5d1cece4579 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:26:22.883004 23866 raft_consensus.cc:3037] T 50ffa350c7fa44fbae946515f401ab9d P ffc9beaa8dfc436492ec2ffef17bb6da [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:26:22.884272 23599 raft_consensus.cc:2454] T 50ffa350c7fa44fbae946515f401ab9d P 3d607a561309454cae15a5d1cece4579 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51b45ca94a0842dda7601ff144fc959d in term 1.
I20260923 17:26:22.884372 23866 raft_consensus.cc:2454] T 50ffa350c7fa44fbae946515f401ab9d P ffc9beaa8dfc436492ec2ffef17bb6da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51b45ca94a0842dda7601ff144fc959d in term 1.
I20260923 17:26:22.884660 23671 leader_election.cc:302] T 50ffa350c7fa44fbae946515f401ab9d P 51b45ca94a0842dda7601ff144fc959d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d607a561309454cae15a5d1cece4579, 51b45ca94a0842dda7601ff144fc959d; no voters:
I20260923 17:26:22.884908 23941 raft_consensus.cc:2781] T 50ffa350c7fa44fbae946515f401ab9d P 51b45ca94a0842dda7601ff144fc959d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:26:22.885154 23941 raft_consensus.cc:686] T 50ffa350c7fa44fbae946515f401ab9d P 51b45ca94a0842dda7601ff144fc959d [term 1 LEADER]: Becoming Leader. State: Replica: 51b45ca94a0842dda7601ff144fc959d, State: Running, Role: LEADER
I20260923 17:26:22.885545 23941 consensus_queue.cc:237] T 50ffa350c7fa44fbae946515f401ab9d P 51b45ca94a0842dda7601ff144fc959d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b45ca94a0842dda7601ff144fc959d" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 38427 } } peers { permanent_uuid: "3d607a561309454cae15a5d1cece4579" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36679 } } peers { permanent_uuid: "ffc9beaa8dfc436492ec2ffef17bb6da" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 43177 } }
I20260923 17:26:22.889355 23429 catalog_manager.cc:5495] T 50ffa350c7fa44fbae946515f401ab9d P 51b45ca94a0842dda7601ff144fc959d reported cstate change: term changed from 0 to 1, leader changed from <none> to 51b45ca94a0842dda7601ff144fc959d (127.11.229.2). New cstate: current_term: 1 leader_uuid: "51b45ca94a0842dda7601ff144fc959d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b45ca94a0842dda7601ff144fc959d" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 38427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d607a561309454cae15a5d1cece4579" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffc9beaa8dfc436492ec2ffef17bb6da" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 43177 } health_report { overall_health: UNKNOWN } } }
W20260923 17:26:22.900019 23916 tablet.cc:2369] T 50ffa350c7fa44fbae946515f401ab9d P ffc9beaa8dfc436492ec2ffef17bb6da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:26:22.919036 23928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:22.913391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48150 (local address 127.11.229.2:38427)
0923 17:26:22.913528 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 17:26:22.913535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:26:22.913552 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:26:22.913801 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:26:22.913805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:26:22.913882 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:26:22.914004 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:26:22.914425 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:22.915242 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:22.916534 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:26:22.916890 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:26:22.916988 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:26:22.917491 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:26:22.918160 (+ 669us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:26:22.918304 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 17:26:22.918814 (+ 510us) server_negotiation.cc:299] Negotiation successful
0923 17:26:22.918882 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 17:26:22.938835 23599 raft_consensus.cc:1261] T 50ffa350c7fa44fbae946515f401ab9d P 3d607a561309454cae15a5d1cece4579 [term 1 FOLLOWER]: Refusing update from remote peer 51b45ca94a0842dda7601ff144fc959d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:26:22.938835 23866 raft_consensus.cc:1261] T 50ffa350c7fa44fbae946515f401ab9d P ffc9beaa8dfc436492ec2ffef17bb6da [term 1 FOLLOWER]: Refusing update from remote peer 51b45ca94a0842dda7601ff144fc959d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:26:22.939697 23941 consensus_queue.cc:1035] T 50ffa350c7fa44fbae946515f401ab9d P 51b45ca94a0842dda7601ff144fc959d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d607a561309454cae15a5d1cece4579" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:26:22.940183 23941 consensus_queue.cc:1035] T 50ffa350c7fa44fbae946515f401ab9d P 51b45ca94a0842dda7601ff144fc959d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffc9beaa8dfc436492ec2ffef17bb6da" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 43177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 17:26:22.944957 23646 tablet.cc:2369] T 50ffa350c7fa44fbae946515f401ab9d P 3d607a561309454cae15a5d1cece4579: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:26:22.964905 23949 mvcc.cc:205] Tried to move back new op lower bound from 7332595232487931904 to 7332595232303255552. Current Snapshot: MvccSnapshot[applied={T|T < 7332595232487931904}]
I20260923 17:26:22.965477 23952 mvcc.cc:205] Tried to move back new op lower bound from 7332595232487931904 to 7332595232303255552. Current Snapshot: MvccSnapshot[applied={T|T < 7332595232487931904}]
I20260923 17:26:22.979769 23429 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:26:22.979993 23429 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:26:22.982918 23429 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c998afe6baf3472b84288a8df1f56bf1: Sending DeleteTablet for 3 replicas of tablet 50ffa350c7fa44fbae946515f401ab9d
I20260923 17:26:22.984301 23714 tablet_service.cc:1514] Processing DeleteTablet for tablet 50ffa350c7fa44fbae946515f401ab9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:26:22 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:48138
I20260923 17:26:22.984843 23561 tablet_service.cc:1514] Processing DeleteTablet for tablet 50ffa350c7fa44fbae946515f401ab9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:26:22 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:51802
I20260923 17:26:22.986033 23849 tablet_service.cc:1514] Processing DeleteTablet for tablet 50ffa350c7fa44fbae946515f401ab9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:26:22 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:52416
I20260923 17:26:22.987123 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 23512
I20260923 17:26:22.989835 23960 tablet_replica.cc:330] T 50ffa350c7fa44fbae946515f401ab9d P ffc9beaa8dfc436492ec2ffef17bb6da: stopping tablet replica
I20260923 17:26:22.990099 23958 tablet_replica.cc:330] T 50ffa350c7fa44fbae946515f401ab9d P 51b45ca94a0842dda7601ff144fc959d: stopping tablet replica
I20260923 17:26:22.990523 23958 raft_consensus.cc:2229] T 50ffa350c7fa44fbae946515f401ab9d P 51b45ca94a0842dda7601ff144fc959d [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:26:22.990888 23960 raft_consensus.cc:2229] T 50ffa350c7fa44fbae946515f401ab9d P ffc9beaa8dfc436492ec2ffef17bb6da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:26:22.991336 23960 raft_consensus.cc:2258] T 50ffa350c7fa44fbae946515f401ab9d P ffc9beaa8dfc436492ec2ffef17bb6da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:26:22.991792 23958 raft_consensus.cc:2258] T 50ffa350c7fa44fbae946515f401ab9d P 51b45ca94a0842dda7601ff144fc959d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:26:22.998028 23958 ts_tablet_manager.cc:1905] T 50ffa350c7fa44fbae946515f401ab9d P 51b45ca94a0842dda7601ff144fc959d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260923 17:26:22.998868 23418 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.229.1:36679 (error 108)
I20260923 17:26:23.000450 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 23651
I20260923 17:26:23.000466 23932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:26:23.000181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:36679 (local address 127.0.0.1:51814)
0923 17:26:23.000301 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:23.000372 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.1:36679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260923 17:26:23.002108 23960 ts_tablet_manager.cc:1905] T 50ffa350c7fa44fbae946515f401ab9d P ffc9beaa8dfc436492ec2ffef17bb6da: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260923 17:26:23.002727 23418 catalog_manager.cc:4584] TS 3d607a561309454cae15a5d1cece4579 (127.11.229.1:36679): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 50ffa350c7fa44fbae946515f401ab9d: Network error: Client connection negotiation failed: client connection to 127.11.229.1:36679: connect: Connection refused (error 111)
I20260923 17:26:23.005044 23960 ts_tablet_manager.cc:1918] T 50ffa350c7fa44fbae946515f401ab9d P ffc9beaa8dfc436492ec2ffef17bb6da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:26:23.005169 23960 log.cc:1196] T 50ffa350c7fa44fbae946515f401ab9d P ffc9beaa8dfc436492ec2ffef17bb6da: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184271597484-12180-0/minicluster-data/ts-2/wal/wals/50ffa350c7fa44fbae946515f401ab9d
I20260923 17:26:23.005546 23960 ts_tablet_manager.cc:1939] T 50ffa350c7fa44fbae946515f401ab9d P ffc9beaa8dfc436492ec2ffef17bb6da: Deleting consensus metadata
I20260923 17:26:23.007061 23416 catalog_manager.cc:4841] TS ffc9beaa8dfc436492ec2ffef17bb6da (127.11.229.3:43177): tablet 50ffa350c7fa44fbae946515f401ab9d (table test-table [id=8424451c9d1a46e4b42ad5fac789c7ee]) successfully deleted
W20260923 17:26:23.016675 23932 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:26:23.016154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:38427 (local address 127.0.0.1:48154)
0923 17:26:23.016275 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0923 17:26:23.016294 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:26:23.016403 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:26:23.016511 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.2:38427: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":34}
I20260923 17:26:23.016752 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 23786
I20260923 17:26:23.029500 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 23399
I20260923 17:26:23.037485 12180 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 17:26:23.372375 12180 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 17:26:23.372495 12180 mini_ranger.cc:66] Stopping Ranger...
I20260923 17:26:23.717720 12180 mini_ranger.cc:68] Stopped Ranger
2026-09-23 17:26:23.717 UTC [22985] LOG: received smart shutdown request
2026-09-23 17:26:23.721 UTC [22985] LOG: background worker "logical replication launcher" (PID 22994) exited with exit code 1
2026-09-23 17:26:23.722 UTC [22989] LOG: shutting down
2026-09-23 17:26:23.911 UTC [22985] LOG: database system is shut down
2026-09-23T17:26:23Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48018 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:26:24 dist-test-slave-ptl9 krb5kdc[23970](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:26:24 dist-test-slave-ptl9 krb5kdc[23970](info): set up 2 sockets
Sep 23 17:26:24 dist-test-slave-ptl9 krb5kdc[23970](info): commencing operation
krb5kdc: starting...
W20260923 17:26:25.065992 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:26:25 dist-test-slave-ptl9 krb5kdc[23970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:26:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:26:25Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.11.229.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.11.229.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.11.229.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/rangeradmin_127.11.229.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.11.229.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/rangeradmin_127.11.229.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.11.229.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.11.229.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.11.229.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/rangerlookup_127.11.229.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.11.229.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/rangerlookup_127.11.229.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.229.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.11.229.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.229.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/HTTP_127.11.229.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.229.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/HTTP_127.11.229.20@KRBTEST.COM.keytab.
I20260923 17:26:25.243952 12180 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task1jLtF9/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/postgres -l logfile start
2026-09-23 17:26:27.776 UTC [24000] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 17:26:27.776 UTC [24000] LOG: listening on IPv4 address "127.11.229.20", port 47651
2026-09-23 17:26:27.779 UTC [24000] LOG: listening on Unix socket "/tmp/.s.PGSQL.47651"
2026-09-23 17:26:27.803 UTC [24003] LOG: database system was shut down at 2026-09-23 17:26:26 UTC
2026-09-23 17:26:27.807 UTC [24000] LOG: database system is ready to accept connections
I20260923 17:26:28.757635 12180 mini_postgres.cc:96] Postgres bound to 47651
2026-09-23 17:26:28.764 UTC [24011] FATAL: database "slave" does not exist
127.11.229.20:47651 - accepting connections
I20260923 17:26:28.764619 12180 mini_ranger.cc:161] Starting Ranger...
I20260923 17:26:28.779721 12180 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 17:26:29.274993 12180 mini_ranger.cc:87] Created ranger Postgres database
I20260923 17:26:29.275066 12180 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ranger-admin
2026-09-23 17:26:29,543 [I] DB FLAVOR :POSTGRES
2026-09-23 17:26:29,544 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 17:26:29,544 [I] Checking connection..
2026-09-23 17:26:29,544 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:47651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 17:26:29,777 [I] Checking connection passed.
2026-09-23 17:26:29,778 [I] --------- Verifying version history table ---------
2026-09-23 17:26:29,778 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:47651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 17:26:30,020 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 17:26:30,021 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:47651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 17:26:30,265 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 17:26:30,265 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 17:26:30,266 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:47651/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 17:26:30.522 UTC [24075] WARNING: there is no transaction in progress
2026-09-23 17:26:30,528 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 17:26:30,529 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:47651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 17:26:30,784 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 17:26:30,784 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 17:26:30,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:47651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 17:26:31,035 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:47651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 17:26:31,266 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:47651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-ptl9.c.gcp-upstream.internal','N') ;"
2026-09-23 17:26:31,487 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:47651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 17:26:31,742 [I] Table x_portal_user does not exist in database ranger
2026-09-23 17:26:31,742 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:47651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 17:26:31,981 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 17:26:31,982 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:47651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 17:26:32,216 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:47651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 17:26:32,455 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 17:26:32,455 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:47651/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 17:26:33.440 UTC [24183] WARNING: there is no transaction in progress
2026-09-23 17:26:33.457 UTC [24183] WARNING: there is no transaction in progress
2026-09-23 17:26:33.474 UTC [24183] WARNING: there is no transaction in progress
2026-09-23 17:26:33.490 UTC [24183] WARNING: there is no transaction in progress
2026-09-23 17:26:33.506 UTC [24183] WARNING: there is no transaction in progress
2026-09-23 17:26:33.524 UTC [24183] WARNING: there is no transaction in progress
2026-09-23 17:26:33.579 UTC [24183] WARNING: there is no transaction in progress
2026-09-23 17:26:33.585 UTC [24183] WARNING: there is no transaction in progress
2026-09-23 17:26:33.599 UTC [24183] WARNING: there is no transaction in progress
2026-09-23 17:26:33.609 UTC [24183] WARNING: there is no transaction in progress
2026-09-23 17:26:33.620 UTC [24183] WARNING: there is no transaction in progress
2026-09-23 17:26:33.633 UTC [24183] WARNING: there is no transaction in progress
2026-09-23 17:26:33.643 UTC [24183] WARNING: there is no transaction in progress
2026-09-23 17:26:33.650 UTC [24183] WARNING: there is no transaction in progress
2026-09-23 17:26:33.658 UTC [24183] WARNING: there is no transaction in progress
2026-09-23 17:26:33.665 UTC [24183] WARNING: there is no transaction in progress
2026-09-23 17:26:34,346 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 17:26:34,346 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:47651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 17:26:34,593 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 17:26:34,593 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:47651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 17:26:34,830 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 17:26:34,830 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:47651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 17:26:35,094 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 17:26:35,094 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:47651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 17:26:35,353 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:47651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 17:26:35,596 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:47651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-ptl9.c.gcp-upstream.internal';"
2026-09-23 17:26:35,844 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 17:26:35,845 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:47651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 17:26:36,073 [I] DB_PATCHES have already been applied
I20260923 17:26:36.079277 12180 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ranger-admin:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task1jLtF9/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 17:26:36.079345 12180 mini_ranger.cc:193] Using host: 127.11.229.20
I20260923 17:26:36.083069 12180 mini_ranger.cc:227] Ranger admin URL: http://127.11.229.20:40957
Sep 23, 2026 5:26:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 5:26:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 5:26:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 5:26:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 5:26:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 5:26:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 5:26:36 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 17:26:37.186771 12180 mini_ranger.cc:160] Time spent starting Ranger: real 8.422s user 0.003s sys 0.007s
Sep 23, 2026 5:26:58 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 5:26:58 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 23, 2026 5:26:58 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 5:26:58 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 5:26:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 5:26:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 5:26:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 5:26:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 5:26:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 5:26:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 5:26:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 5:26:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 5:26:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 5:26:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 5:26:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 5:26:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 5:26:58 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 5:26:59 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 17:26:59.584743 12180 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.11.229.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.11.229.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.11.229.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/rangerkms_127.11.229.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.11.229.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/rangerkms_127.11.229.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.229.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.11.229.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.229.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/HTTP_127.11.229.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.229.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/HTTP_127.11.229.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 17:26:59 dist-test-slave-ptl9 krb5kdc[23970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184419, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 17:26:59.673202 12180 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 17:26:59.683137 12180 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 17:27:00.649580 12180 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 17:27:00.649684 12180 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ranger-kms
I20260923 17:27:00.649706 12180 mini_ranger_kms.cc:227] Using postgres at 127.11.229.20:47651
2026-09-23 17:27:00,816 [I] DB FLAVOR :POSTGRES
2026-09-23 17:27:00,816 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 17:27:00,817 [I] Checking connection
2026-09-23 17:27:00,817 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:47651/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 17:27:01,049 [I] connection success
2026-09-23 17:27:01,049 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 17:27:01,049 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 17:27:01,049 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:47651/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 17:27:01,280 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 17:27:01,280 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 17:27:01,280 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 17:27:01,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.229.20:47651/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 17:27:01,563 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 17:27:02.095046 12180 mini_ranger_kms.cc:310] Created kms service
I20260923 17:27:02.270673 12180 mini_ranger_kms.cc:326] Created kudu user
I20260923 17:27:02.321878 12180 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 17:27:02 dist-test-slave-ptl9 krb5kdc[23970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184419, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.11.229.20@KRBTEST.COM
I20260923 17:27:02.732453 12180 mini_ranger_kms.cc:384] Added ranger policy
I20260923 17:27:02.732607 12180 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ranger-kms:/tmp/dist-test-task1jLtF9/build/debug/bin/postgresql.jar:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task1jLtF9/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 17:27:02.732630 12180 mini_ranger_kms.cc:240] Using host: 127.11.229.20
I20260923 17:27:02.736943 12180 mini_ranger_kms.cc:276] Ranger KMS PID: 24363
I20260923 17:27:02.737035 12180 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.11.229.20:60021
Sep 23, 2026 5:27:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 5:27:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 5:27:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 5:27:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 5:27:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 5:27:04 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260923 17:27:04.979736 12180 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.307s user 0.003s sys 0.148s
I20260923 17:27:04.979878 12180 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 17:27:04.979967 12180 mini_ranger_kms.cc:401] 127.11.229.20:60021/kms/v1/keys
Sep 23 17:27:05 dist-test-slave-ptl9 krb5kdc[23970](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790184425, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.229.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 23 17:27:05 dist-test-slave-ptl9 krb5kdc[23970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184425, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.229.20@KRBTEST.COM for HTTP/127.11.229.20@KRBTEST.COM
Sep 23 17:27:05 dist-test-slave-ptl9 krb5kdc[23970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184425, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.229.20@KRBTEST.COM for HTTP/127.11.229.20@KRBTEST.COM
Sep 23 17:27:08 dist-test-slave-ptl9 krb5kdc[23970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.11.229.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
I20260923 17:27:08.265427 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:42581
--webserver_interface=127.11.229.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:42581
--ranger_config_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.229.20:60021/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:27:08.384737 24414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:08.385040 24414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:08.385082 24414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:08.389328 24414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:27:08.389413 24414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:08.389433 24414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:08.389451 24414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:27:08.389467 24414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:27:08.395445 24414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/master-0/wal
--ranger_kms_url=127.11.229.20:60021/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:42581
--ranger_config_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:42581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:27:08.396703 24414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:08.397898 24414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:08.407650 24419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:08.407768 24420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:08.407768 24422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:08.407668 24414 server_base.cc:1034] running on GCE node
I20260923 17:27:08.408685 24414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:08.410074 24414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:08.411299 24414 hybrid_clock.cc:648] HybridClock initialized: now 1790184428411283 us; error 32 us; skew 500 ppm
Sep 23 17:27:08 dist-test-slave-ptl9 krb5kdc[23970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184428, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:08.414521 24414 init.cc:377] Logged in from keytab as kudu/127.11.229.62@KRBTEST.COM (short username kudu)
I20260923 17:27:08.415548 24414 webserver.cc:460] Webserver started at http://127.11.229.62:33733/ using document root <none> and password file <none>
I20260923 17:27:08.416181 24414 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:08.416256 24414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:08.416492 24414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:27:08 dist-test-slave-ptl9 krb5kdc[23970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184428, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for HTTP/127.11.229.20@KRBTEST.COM
I20260923 17:27:08.469897 24414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "a5b00a58242c45f6b707c53b9b06ad0f"
format_stamp: "Formatted at 2026-09-23 17:27:08 on dist-test-slave-ptl9"
server_key: "d43251f6820a3ac583ef1c22422717ee"
server_key_iv: "4bd02983ba58ef662922ea407dff30c5"
server_key_version: "kuduclusterkey@0"
I20260923 17:27:08.470882 24414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "a5b00a58242c45f6b707c53b9b06ad0f"
format_stamp: "Formatted at 2026-09-23 17:27:08 on dist-test-slave-ptl9"
server_key: "d43251f6820a3ac583ef1c22422717ee"
server_key_iv: "4bd02983ba58ef662922ea407dff30c5"
server_key_version: "kuduclusterkey@0"
I20260923 17:27:08.475497 24414 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:27:08.493573 24429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:08.495103 24414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:27:08.495450 24414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "a5b00a58242c45f6b707c53b9b06ad0f"
format_stamp: "Formatted at 2026-09-23 17:27:08 on dist-test-slave-ptl9"
server_key: "d43251f6820a3ac583ef1c22422717ee"
server_key_iv: "4bd02983ba58ef662922ea407dff30c5"
server_key_version: "kuduclusterkey@0"
I20260923 17:27:08.495615 24414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:08.519616 24414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:08.522805 24414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:08.523015 24414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:08.530598 24414 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:42581
I20260923 17:27:08.530624 24481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:42581 every 8 connection(s)
I20260923 17:27:08.531602 24414 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:27:08.534205 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 24414
I20260923 17:27:08.534329 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:27:08.534654 12180 external_mini_cluster.cc:1428] Setting key fe187bdca82010efa9c53608680d3dc4
I20260923 17:27:08.534675 24482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:27:08.541052 24482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f: Bootstrap starting.
I20260923 17:27:08.543329 24482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f: Neither blocks nor log segments found. Creating new log.
Sep 23 17:27:08 dist-test-slave-ptl9 krb5kdc[23970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:27:08.544158 24482 log.cc:824] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f: Log is configured to *not* fsync() on all Append() calls
I20260923 17:27:08.546855 24482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f: No bootstrap required, opened a new log
I20260923 17:27:08.550410 24485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:08.536150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42660 (local address 127.11.229.62:42581)
0923 17:27:08.536630 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:08.536646 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:08.536684 (+ 38us) server_negotiation.cc:407] Connection header received
0923 17:27:08.537560 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:08.537581 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:08.537922 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:08.538259 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:08.539078 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:08.540115 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:08.540985 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:08.541559 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:08.541735 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:08.544329 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:08.544337 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:08.544360 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:08.544390 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:08.547131 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:08.547581 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:08.547587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:08.547594 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:08.547729 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:08.548010 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:08.548013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:08.548015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:08.548533 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:08.548737 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:08.549043 (+ 306us) server_negotiation.cc:299] Negotiation successful
0923 17:27:08.549380 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":327,"thread_start_us":142,"threads_started":1}
I20260923 17:27:08.551019 24482 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b00a58242c45f6b707c53b9b06ad0f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 42581 } }
I20260923 17:27:08.551297 24482 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:27:08.551353 24482 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b00a58242c45f6b707c53b9b06ad0f, State: Initialized, Role: FOLLOWER
I20260923 17:27:08.551826 24482 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b00a58242c45f6b707c53b9b06ad0f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 42581 } }
I20260923 17:27:08.551949 24482 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:27:08.552032 24482 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:27:08.552132 24482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:27:08.553355 24482 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b00a58242c45f6b707c53b9b06ad0f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 42581 } }
I20260923 17:27:08.553710 24482 leader_election.cc:302] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5b00a58242c45f6b707c53b9b06ad0f; no voters:
I20260923 17:27:08.554040 24482 leader_election.cc:288] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:27:08.554193 24487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:27:08.554446 24487 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f [term 1 LEADER]: Becoming Leader. State: Replica: a5b00a58242c45f6b707c53b9b06ad0f, State: Running, Role: LEADER
I20260923 17:27:08.554805 24487 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b00a58242c45f6b707c53b9b06ad0f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 42581 } }
I20260923 17:27:08.555305 24482 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:27:08.556615 24489 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5b00a58242c45f6b707c53b9b06ad0f. Latest consensus state: current_term: 1 leader_uuid: "a5b00a58242c45f6b707c53b9b06ad0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b00a58242c45f6b707c53b9b06ad0f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 42581 } } }
I20260923 17:27:08.556751 24489 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:08.557134 24488 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5b00a58242c45f6b707c53b9b06ad0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b00a58242c45f6b707c53b9b06ad0f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 42581 } } }
I20260923 17:27:08.557255 24488 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:08.557621 24496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:27:08.557809 24482 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 17:27:08.563427 24496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:27:08.571367 24496 catalog_manager.cc:1347] Generated new cluster ID: 9f9f23f601554f89a3c26009e4696af0
I20260923 17:27:08.571516 24496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:27:08.602705 24496 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:27:08.604223 24496 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:27:08.610761 24496 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f: Generated new TSK 0
I20260923 17:27:08.611742 24496 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.229.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab.
Entry for principal kudu/127.11.229.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab.
I20260923 17:27:09.328886 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.229.20:60021/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.1:0
--local_ip_for_outbound_sockets=127.11.229.1
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:42581
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:27:09.482558 24527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:09.482903 24527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:09.483011 24527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:09.487403 24527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:09.487497 24527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:09.487603 24527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.1
I20260923 17:27:09.494067 24527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.11.229.20:60021/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:42581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.1
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:27:09.495625 24527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:09.497331 24527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:09.509938 24534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:09.511121 24537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:09.511317 24535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:09.512588 24527 server_base.cc:1034] running on GCE node
I20260923 17:27:09.513178 24527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:09.514079 24527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:09.515339 24527 hybrid_clock.cc:648] HybridClock initialized: now 1790184429515304 us; error 49 us; skew 500 ppm
Sep 23 17:27:09 dist-test-slave-ptl9 krb5kdc[23970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184429, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:09.519061 24527 init.cc:377] Logged in from keytab as kudu/127.11.229.1@KRBTEST.COM (short username kudu)
I20260923 17:27:09.520267 24527 webserver.cc:460] Webserver started at http://127.11.229.1:37225/ using document root <none> and password file <none>
I20260923 17:27:09.521093 24527 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:09.521247 24527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:09.521585 24527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:27:09 dist-test-slave-ptl9 krb5kdc[23970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184429, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for HTTP/127.11.229.20@KRBTEST.COM
I20260923 17:27:09.538659 24527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-0/data/instance:
uuid: "04bb5ca847f84da29571b62429b9b56c"
format_stamp: "Formatted at 2026-09-23 17:27:09 on dist-test-slave-ptl9"
server_key: "f62cd04792167639ea550bcbcdb85a1c"
server_key_iv: "1000e96fc71d435a0bf0cb0d8286e1c8"
server_key_version: "kuduclusterkey@0"
I20260923 17:27:09.539644 24527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance:
uuid: "04bb5ca847f84da29571b62429b9b56c"
format_stamp: "Formatted at 2026-09-23 17:27:09 on dist-test-slave-ptl9"
server_key: "f62cd04792167639ea550bcbcdb85a1c"
server_key_iv: "1000e96fc71d435a0bf0cb0d8286e1c8"
server_key_version: "kuduclusterkey@0"
I20260923 17:27:09.544878 24527 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 17:27:09.565443 24544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:09.567245 24527 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:27:09.567384 24527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-0/wal
uuid: "04bb5ca847f84da29571b62429b9b56c"
format_stamp: "Formatted at 2026-09-23 17:27:09 on dist-test-slave-ptl9"
server_key: "f62cd04792167639ea550bcbcdb85a1c"
server_key_iv: "1000e96fc71d435a0bf0cb0d8286e1c8"
server_key_version: "kuduclusterkey@0"
I20260923 17:27:09.567517 24527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:09.591480 24527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:09.595511 24527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:09.595780 24527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:09.596539 24527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:27:09.597698 24527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:27:09.597796 24527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:09.597886 24527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:27:09.597937 24527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:09.608381 24527 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.1:40761
I20260923 17:27:09.609642 24527 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
I20260923 17:27:09.615841 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 24527
I20260923 17:27:09.615972 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance
I20260923 17:27:09.616341 12180 external_mini_cluster.cc:1428] Setting key dc06fa6db83c5c13c07f21e1e7927036
I20260923 17:27:09.616989 24657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.1:40761 every 8 connection(s)
WARNING: no policy specified for kudu/127.11.229.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.2@KRBTEST.COM" created.
Sep 23 17:27:09 dist-test-slave-ptl9 krb5kdc[23970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184429, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:27:09.639705 24660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:09.619474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:42581 (local address 127.11.229.1:34393)
0923 17:27:09.620773 (+ 1299us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:09.620838 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:09.622145 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:09.622974 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:09.622985 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:09.623374 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:09.624123 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:09.624147 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:09.625509 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:09.625513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:09.627038 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:09.627050 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:09.627883 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:09.627888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:09.628094 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:09.628773 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:09.628795 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:09.632912 (+ 4117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:09.635510 (+ 2598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:09.635520 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:09.635545 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:09.635971 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:09.636331 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:09.636337 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:09.636341 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:09.636552 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:09.637200 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:09.637209 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:09.637677 (+ 468us) client_negotiation.cc:769] Sending connection context
0923 17:27:09.637933 (+ 256us) client_negotiation.cc:240] Negotiation successful
0923 17:27:09.638357 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1043,"thread_start_us":160,"threads_started":1}
I20260923 17:27:09.640875 24662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:09.620064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:34393 (local address 127.11.229.62:42581)
0923 17:27:09.620330 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:09.620335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:09.620937 (+ 602us) server_negotiation.cc:407] Connection header received
0923 17:27:09.622396 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:09.622402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:09.622467 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:09.622531 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:09.624319 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:09.625327 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:09.627189 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:09.627564 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:09.627693 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:09.633212 (+ 5519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:09.633220 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:09.633226 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:09.633289 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:09.635377 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:09.636108 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:09.636114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:09.636119 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:09.636220 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:09.636683 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:09.636689 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:09.636694 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:09.636980 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:09.637097 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:09.640367 (+ 3270us) server_negotiation.cc:299] Negotiation successful
0923 17:27:09.640536 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":101,"threads_started":1}
I20260923 17:27:09.642292 24658 heartbeater.cc:347] Connected to a master server at 127.11.229.62:42581
I20260923 17:27:09.642576 24658 heartbeater.cc:464] Registering TS with master...
I20260923 17:27:09.643245 24658 heartbeater.cc:511] Master 127.11.229.62:42581 requested a full tablet report, sending...
I20260923 17:27:09.645351 24446 ts_manager.cc:194] Registered new tserver with Master: 04bb5ca847f84da29571b62429b9b56c (127.11.229.1:40761)
I20260923 17:27:09.646961 24446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.1@KRBTEST.COM'} at 127.11.229.1:34393
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab.
Entry for principal kudu/127.11.229.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab.
I20260923 17:27:09.657809 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.229.20:60021/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.2:0
--local_ip_for_outbound_sockets=127.11.229.2
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:42581
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:27:09 dist-test-slave-ptl9 krb5kdc[23970](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790184429, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260923 17:27:09.819206 24666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:09.819676 24666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:09.819761 24666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:09.825932 24666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:09.826124 24666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:09.826249 24666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.2
I20260923 17:27:09.832775 24666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.11.229.20:60021/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:42581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.2
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:27:09.834111 24666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:09.835398 24666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:09.846918 24673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:09.847040 24672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:09.846918 24675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:09.848273 24666 server_base.cc:1034] running on GCE node
I20260923 17:27:09.848863 24666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:09.849735 24666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:09.850932 24666 hybrid_clock.cc:648] HybridClock initialized: now 1790184429850895 us; error 51 us; skew 500 ppm
Sep 23 17:27:09 dist-test-slave-ptl9 krb5kdc[23970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184429, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:09.854768 24666 init.cc:377] Logged in from keytab as kudu/127.11.229.2@KRBTEST.COM (short username kudu)
I20260923 17:27:09.856127 24666 webserver.cc:460] Webserver started at http://127.11.229.2:38867/ using document root <none> and password file <none>
I20260923 17:27:09.857009 24666 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:09.857174 24666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:09.857525 24666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:27:09 dist-test-slave-ptl9 krb5kdc[23970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184429, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for HTTP/127.11.229.20@KRBTEST.COM
I20260923 17:27:09.874647 24666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-1/data/instance:
uuid: "978632d31add4f05ad4a37d8b1883d51"
format_stamp: "Formatted at 2026-09-23 17:27:09 on dist-test-slave-ptl9"
server_key: "f41e73e4b9a002dc7ef1b2d563aab668"
server_key_iv: "ac19df5643d81242b8eb4c7cf614e8d2"
server_key_version: "kuduclusterkey@0"
I20260923 17:27:09.875473 24666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance:
uuid: "978632d31add4f05ad4a37d8b1883d51"
format_stamp: "Formatted at 2026-09-23 17:27:09 on dist-test-slave-ptl9"
server_key: "f41e73e4b9a002dc7ef1b2d563aab668"
server_key_iv: "ac19df5643d81242b8eb4c7cf614e8d2"
server_key_version: "kuduclusterkey@0"
I20260923 17:27:09.880555 24666 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 17:27:09.899138 24682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:09.901060 24666 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260923 17:27:09.901304 24666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-1/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-1/wal
uuid: "978632d31add4f05ad4a37d8b1883d51"
format_stamp: "Formatted at 2026-09-23 17:27:09 on dist-test-slave-ptl9"
server_key: "f41e73e4b9a002dc7ef1b2d563aab668"
server_key_iv: "ac19df5643d81242b8eb4c7cf614e8d2"
server_key_version: "kuduclusterkey@0"
I20260923 17:27:09.901538 24666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:09.950804 24666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:09.953962 24666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:09.954164 24666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:09.954798 24666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:27:09.955760 24666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:27:09.955829 24666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:09.955899 24666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:27:09.955946 24666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:09.964955 24666 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.2:36279
I20260923 17:27:09.965020 24795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.2:36279 every 8 connection(s)
I20260923 17:27:09.966073 24666 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
I20260923 17:27:09.968766 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 24666
I20260923 17:27:09.968904 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance
I20260923 17:27:09.969216 12180 external_mini_cluster.cc:1428] Setting key de3459ce938a28f654db98ff49809c42
WARNING: no policy specified for kudu/127.11.229.3@KRBTEST.COM; defaulting to no policy
Sep 23 17:27:09 dist-test-slave-ptl9 krb5kdc[23970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184429, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.3@KRBTEST.COM" created.
I20260923 17:27:09.996205 24662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:09.968624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:34711 (local address 127.11.229.62:42581)
0923 17:27:09.968817 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:09.968824 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:09.970443 (+ 1619us) server_negotiation.cc:407] Connection header received
0923 17:27:09.971909 (+ 1466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:09.971915 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:09.971982 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:09.972078 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:09.974034 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:09.974729 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:09.984815 (+ 10086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:09.985215 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:09.985365 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:09.988731 (+ 3366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:09.988735 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:09.988738 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:09.988772 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:09.990894 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:09.993908 (+ 3014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:09.993914 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:09.993916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:09.994056 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:09.994622 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:09.994626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:09.994628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:09.994910 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:09.995005 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:09.995838 (+ 833us) server_negotiation.cc:299] Negotiation successful
0923 17:27:09.996017 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 17:27:09.997469 24798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:09.969161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:42581 (local address 127.11.229.2:34711)
0923 17:27:09.970239 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:09.970313 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:09.971603 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:09.972346 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:09.972358 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:09.972807 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:09.973745 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:09.973772 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:09.974869 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:09.974873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:09.984547 (+ 9674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:09.984576 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:09.985630 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:09.985634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:09.985892 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:09.986662 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:09.986688 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:09.988437 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:09.991061 (+ 2624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:09.991070 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:09.991091 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:09.991499 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:09.994205 (+ 2706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:09.994208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:09.994210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:09.994487 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:09.995085 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:09.995093 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:09.995487 (+ 394us) client_negotiation.cc:769] Sending connection context
0923 17:27:09.995720 (+ 233us) client_negotiation.cc:240] Negotiation successful
0923 17:27:09.996137 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":160,"threads_started":1}
I20260923 17:27:09.999141 24796 heartbeater.cc:347] Connected to a master server at 127.11.229.62:42581
I20260923 17:27:09.999470 24796 heartbeater.cc:464] Registering TS with master...
I20260923 17:27:10.000263 24796 heartbeater.cc:511] Master 127.11.229.62:42581 requested a full tablet report, sending...
I20260923 17:27:10.001703 24446 ts_manager.cc:194] Registered new tserver with Master: 978632d31add4f05ad4a37d8b1883d51 (127.11.229.2:36279)
I20260923 17:27:10.002528 24446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.2@KRBTEST.COM'} at 127.11.229.2:34711
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab.
Entry for principal kudu/127.11.229.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab.
I20260923 17:27:10.021664 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.229.20:60021/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.3:0
--local_ip_for_outbound_sockets=127.11.229.3
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:42581
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:27:10 dist-test-slave-ptl9 krb5kdc[23970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184429, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for HTTP/127.11.229.20@KRBTEST.COM
W20260923 17:27:10.195655 24801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:10.196092 24801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:10.196211 24801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:10.202271 24801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:10.202478 24801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:10.202611 24801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.3
I20260923 17:27:10.211320 24801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.11.229.20:60021/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:42581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.3
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:27:10.213160 24801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:10.214723 24801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:10.227731 24807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:10.227934 24809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:10.227767 24806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:10.228821 24801 server_base.cc:1034] running on GCE node
I20260923 17:27:10.229373 24801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:10.230202 24801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:10.231400 24801 hybrid_clock.cc:648] HybridClock initialized: now 1790184430231372 us; error 39 us; skew 500 ppm
Sep 23 17:27:10 dist-test-slave-ptl9 krb5kdc[23970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184430, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:10.235163 24801 init.cc:377] Logged in from keytab as kudu/127.11.229.3@KRBTEST.COM (short username kudu)
I20260923 17:27:10.236289 24801 webserver.cc:460] Webserver started at http://127.11.229.3:44173/ using document root <none> and password file <none>
I20260923 17:27:10.237020 24801 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:10.237097 24801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:10.237344 24801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:27:10 dist-test-slave-ptl9 krb5kdc[23970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184430, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for HTTP/127.11.229.20@KRBTEST.COM
I20260923 17:27:10.256392 24801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-2/data/instance:
uuid: "0c88a5d0eb1d4fd2b580c79e883ef210"
format_stamp: "Formatted at 2026-09-23 17:27:10 on dist-test-slave-ptl9"
server_key: "b8d2db4ad8865567cd02329b322e7fde"
server_key_iv: "dffe4f6e61cf4eacd3d4e031b0b5ca9f"
server_key_version: "kuduclusterkey@0"
I20260923 17:27:10.257768 24801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance:
uuid: "0c88a5d0eb1d4fd2b580c79e883ef210"
format_stamp: "Formatted at 2026-09-23 17:27:10 on dist-test-slave-ptl9"
server_key: "b8d2db4ad8865567cd02329b322e7fde"
server_key_iv: "dffe4f6e61cf4eacd3d4e031b0b5ca9f"
server_key_version: "kuduclusterkey@0"
I20260923 17:27:10.263236 24801 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 17:27:10.279631 24816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:10.281246 24801 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:27:10.281446 24801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-2/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-2/wal
uuid: "0c88a5d0eb1d4fd2b580c79e883ef210"
format_stamp: "Formatted at 2026-09-23 17:27:10 on dist-test-slave-ptl9"
server_key: "b8d2db4ad8865567cd02329b322e7fde"
server_key_iv: "dffe4f6e61cf4eacd3d4e031b0b5ca9f"
server_key_version: "kuduclusterkey@0"
I20260923 17:27:10.281603 24801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:10.297863 24801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:10.301671 24801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:10.301945 24801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:10.302825 24801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:27:10.303915 24801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:27:10.304013 24801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:10.304114 24801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:27:10.304183 24801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:10.314549 24801 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.3:38155
I20260923 17:27:10.318943 24801 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
I20260923 17:27:10.323957 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 24801
I20260923 17:27:10.324086 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance
I20260923 17:27:10.324415 12180 external_mini_cluster.cc:1428] Setting key 92f8f160f2ac7f4de72818b1180455f4
I20260923 17:27:10.320803 24929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.3:38155 every 8 connection(s)
Sep 23 17:27:10 dist-test-slave-ptl9 krb5kdc[23970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184430, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:27:10.345206 24662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:10.323028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:39715 (local address 127.11.229.62:42581)
0923 17:27:10.323164 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:10.323168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:10.329785 (+ 6617us) server_negotiation.cc:407] Connection header received
0923 17:27:10.331014 (+ 1229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:10.331018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:10.331072 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:10.331120 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:10.332804 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:10.333669 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:10.335347 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:10.335682 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:10.335767 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:10.340480 (+ 4713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:10.340485 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:10.340488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:10.340522 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:10.342491 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:10.343182 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:10.343187 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:10.343192 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:10.343296 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:10.343809 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:10.343813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:10.343814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:10.344054 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:10.344131 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:10.344892 (+ 761us) server_negotiation.cc:299] Negotiation successful
0923 17:27:10.345033 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 17:27:10.347208 24932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:10.323565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:42581 (local address 127.11.229.3:39715)
0923 17:27:10.329620 (+ 6055us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:10.329681 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:10.330763 (+ 1082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:10.331557 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:10.331567 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:10.331939 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:10.332625 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:10.332643 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:10.333859 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:10.333862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:10.335210 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:10.335222 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:10.335979 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:10.335983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:10.336179 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:10.338481 (+ 2302us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:10.338502 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:10.340250 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:10.342641 (+ 2391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:10.342648 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:10.342667 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:10.343032 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:10.343431 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:10.343435 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:10.343439 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:10.343647 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:10.344228 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:10.344235 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:10.344587 (+ 352us) client_negotiation.cc:769] Sending connection context
0923 17:27:10.345735 (+ 1148us) client_negotiation.cc:240] Negotiation successful
0923 17:27:10.346105 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5795,"thread_start_us":160,"threads_started":1}
I20260923 17:27:10.350458 24930 heartbeater.cc:347] Connected to a master server at 127.11.229.62:42581
I20260923 17:27:10.350723 24930 heartbeater.cc:464] Registering TS with master...
I20260923 17:27:10.351334 24930 heartbeater.cc:511] Master 127.11.229.62:42581 requested a full tablet report, sending...
I20260923 17:27:10.352578 24446 ts_manager.cc:194] Registered new tserver with Master: 0c88a5d0eb1d4fd2b580c79e883ef210 (127.11.229.3:38155)
I20260923 17:27:10.353348 24446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.3@KRBTEST.COM'} at 127.11.229.3:39715
I20260923 17:27:10.364068 12180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:27:10.364918 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 24527
I20260923 17:27:10.375128 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 24666
I20260923 17:27:10.384589 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 24801
I20260923 17:27:10.393421 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 24414
I20260923 17:27:10.400257 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:42581
--webserver_interface=127.11.229.62
--webserver_port=33733
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:42581
--ranger_config_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.229.20:60021/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.229.20:60021/kms with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:27:10.518293 24933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:10.518553 24933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:10.518594 24933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:10.522644 24933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:27:10.522701 24933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:10.522722 24933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:10.522737 24933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:27:10.522787 24933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:27:10.528177 24933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.229.20:60021/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:42581
--ranger_config_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:42581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=33733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:27:10.529193 24933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:10.530224 24933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:10.538887 24939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:10.538919 24933 server_base.cc:1034] running on GCE node
W20260923 17:27:10.538877 24938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:10.538877 24941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:10.539721 24933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:10.540791 24933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:10.541946 24933 hybrid_clock.cc:648] HybridClock initialized: now 1790184430541926 us; error 26 us; skew 500 ppm
Sep 23 17:27:10 dist-test-slave-ptl9 krb5kdc[23970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184430, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:10.544625 24933 init.cc:377] Logged in from keytab as kudu/127.11.229.62@KRBTEST.COM (short username kudu)
I20260923 17:27:10.545531 24933 webserver.cc:460] Webserver started at http://127.11.229.62:33733/ using document root <none> and password file <none>
I20260923 17:27:10.546069 24933 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:10.546134 24933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 17:27:10 dist-test-slave-ptl9 krb5kdc[23970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184430, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for HTTP/127.11.229.20@KRBTEST.COM
I20260923 17:27:10.609809 24933 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:27:10.612702 24949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:10.614414 24933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:27:10.614570 24933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "a5b00a58242c45f6b707c53b9b06ad0f"
format_stamp: "Formatted at 2026-09-23 17:27:08 on dist-test-slave-ptl9"
server_key: "d43251f6820a3ac583ef1c22422717ee"
server_key_iv: "4bd02983ba58ef662922ea407dff30c5"
server_key_version: "kuduclusterkey@0"
I20260923 17:27:10.615310 24933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:10.635524 24933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:10.638260 24933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:10.638417 24933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:10.645710 24933 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:42581
I20260923 17:27:10.645717 25001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:42581 every 8 connection(s)
I20260923 17:27:10.646819 24933 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:27:10.648223 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 24933
I20260923 17:27:10.648743 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.229.20:60021/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.1:40761
--local_ip_for_outbound_sockets=127.11.229.1
--tserver_master_addrs=127.11.229.62:42581
--webserver_port=37225
--webserver_interface=127.11.229.1
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.229.20:60021/kms with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:27:10.652192 25002 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:27:10.654026 25002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f: Bootstrap starting.
I20260923 17:27:10.670672 25002 log.cc:824] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f: Log is configured to *not* fsync() on all Append() calls
I20260923 17:27:10.677515 25002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:27:10.677928 25002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f: Bootstrap complete.
I20260923 17:27:10.682266 25002 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b00a58242c45f6b707c53b9b06ad0f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 42581 } }
I20260923 17:27:10.682662 25002 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b00a58242c45f6b707c53b9b06ad0f, State: Initialized, Role: FOLLOWER
I20260923 17:27:10.683207 25002 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b00a58242c45f6b707c53b9b06ad0f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 42581 } }
I20260923 17:27:10.683341 25002 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:27:10.683429 25002 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:27:10.683542 25002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f [term 1 FOLLOWER]: Advancing to term 2
I20260923 17:27:10.685590 25002 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b00a58242c45f6b707c53b9b06ad0f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 42581 } }
I20260923 17:27:10.685936 25002 leader_election.cc:302] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5b00a58242c45f6b707c53b9b06ad0f; no voters:
I20260923 17:27:10.686240 25002 leader_election.cc:288] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 17:27:10.686388 25006 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f [term 2 FOLLOWER]: Leader election won for term 2
I20260923 17:27:10.686661 25006 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f [term 2 LEADER]: Becoming Leader. State: Replica: a5b00a58242c45f6b707c53b9b06ad0f, State: Running, Role: LEADER
I20260923 17:27:10.687043 25006 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b00a58242c45f6b707c53b9b06ad0f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 42581 } }
I20260923 17:27:10.687471 25002 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:27:10.688606 25008 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5b00a58242c45f6b707c53b9b06ad0f. Latest consensus state: current_term: 2 leader_uuid: "a5b00a58242c45f6b707c53b9b06ad0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b00a58242c45f6b707c53b9b06ad0f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 42581 } } }
I20260923 17:27:10.688726 25008 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:10.688968 25007 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a5b00a58242c45f6b707c53b9b06ad0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b00a58242c45f6b707c53b9b06ad0f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 42581 } } }
I20260923 17:27:10.689044 25007 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:10.689083 25015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:27:10.692322 25015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:27:10.692813 25002 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 17:27:10.693008 25015 catalog_manager.cc:1259] Loaded cluster ID: 9f9f23f601554f89a3c26009e4696af0
I20260923 17:27:10.693075 25015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:27:10.695808 25015 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:27:10.696480 25015 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f: Loaded TSK: 0
I20260923 17:27:10.697769 25015 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 17:27:10.792979 25004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:10.793483 25004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:10.793627 25004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:10.800361 25004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:10.800443 25004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:10.800539 25004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.1
I20260923 17:27:10.808626 25004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.229.20:60021/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.1:40761
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.229.1
--webserver_port=37225
--tserver_master_addrs=127.11.229.62:42581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.1
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:27:10.810140 25004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:10.811498 25004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:10.824383 25042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:10.824383 25040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:10.824478 25039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:10.825001 25004 server_base.cc:1034] running on GCE node
I20260923 17:27:10.825449 25004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:10.826164 25004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:10.829440 25004 hybrid_clock.cc:648] HybridClock initialized: now 1790184430828791 us; error 663 us; skew 500 ppm
Sep 23 17:27:10 dist-test-slave-ptl9 krb5kdc[23970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184430, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:10.832518 25004 init.cc:377] Logged in from keytab as kudu/127.11.229.1@KRBTEST.COM (short username kudu)
I20260923 17:27:10.833632 25004 webserver.cc:460] Webserver started at http://127.11.229.1:37225/ using document root <none> and password file <none>
I20260923 17:27:10.834236 25004 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:10.834339 25004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 17:27:10 dist-test-slave-ptl9 krb5kdc[23970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184430, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for HTTP/127.11.229.20@KRBTEST.COM
I20260923 17:27:10.861665 25004 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.003s sys 0.000s
I20260923 17:27:10.866415 25050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:10.868218 25004 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260923 17:27:10.868402 25004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-0/wal
uuid: "04bb5ca847f84da29571b62429b9b56c"
format_stamp: "Formatted at 2026-09-23 17:27:09 on dist-test-slave-ptl9"
server_key: "f62cd04792167639ea550bcbcdb85a1c"
server_key_iv: "1000e96fc71d435a0bf0cb0d8286e1c8"
server_key_version: "kuduclusterkey@0"
I20260923 17:27:10.869274 25004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:10.886004 25004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:10.889739 25004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:10.890945 25004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:10.891760 25004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:27:10.892969 25004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:27:10.893036 25004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:10.893106 25004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:27:10.893152 25004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:10.904935 25004 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.1:40761
I20260923 17:27:10.904951 25163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.1:40761 every 8 connection(s)
I20260923 17:27:10.906379 25004 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
I20260923 17:27:10.906976 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 25004
I20260923 17:27:10.907480 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.229.20:60021/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.2:36279
--local_ip_for_outbound_sockets=127.11.229.2
--tserver_master_addrs=127.11.229.62:42581
--webserver_port=38867
--webserver_interface=127.11.229.2
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.229.20:60021/kms with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:27:10 dist-test-slave-ptl9 krb5kdc[23970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184430, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:27:10.936754 25168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:10.913770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:59965 (local address 127.11.229.62:42581)
0923 17:27:10.916127 (+ 2357us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:10.916151 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:10.916197 (+ 46us) server_negotiation.cc:407] Connection header received
0923 17:27:10.916508 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:10.916523 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:10.916949 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:10.917337 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:10.919241 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:10.921029 (+ 1788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:10.922915 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:10.923692 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:10.923856 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:10.927160 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:10.927167 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:10.927184 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:10.927229 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:10.929081 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:10.929828 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:10.929834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:10.929841 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:10.929988 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:10.933784 (+ 3796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:10.933788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:10.933790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:10.934316 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:10.934441 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:10.935333 (+ 892us) server_negotiation.cc:299] Negotiation successful
0923 17:27:10.935667 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2187,"thread_start_us":110,"threads_started":1}
I20260923 17:27:10.938803 25167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:10.912128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:42581 (local address 127.11.229.1:59965)
0923 17:27:10.914461 (+ 2333us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:10.914517 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:10.915618 (+ 1101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:10.917842 (+ 2224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:10.917851 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:10.918205 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:10.919012 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:10.919032 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:10.921203 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:10.921207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:10.922728 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:10.922739 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:10.924095 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:10.924100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:10.924315 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:10.925033 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:10.925057 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:10.926919 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:10.929233 (+ 2314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:10.929240 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:10.929279 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:10.929661 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:10.933419 (+ 3758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:10.933423 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:10.933425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:10.933623 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:10.934542 (+ 919us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:10.934549 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:10.934931 (+ 382us) client_negotiation.cc:769] Sending connection context
0923 17:27:10.937595 (+ 2664us) client_negotiation.cc:240] Negotiation successful
0923 17:27:10.937927 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2086,"thread_start_us":145,"threads_started":1}
I20260923 17:27:10.941864 25164 heartbeater.cc:347] Connected to a master server at 127.11.229.62:42581
I20260923 17:27:10.942096 25164 heartbeater.cc:464] Registering TS with master...
I20260923 17:27:10.942651 25164 heartbeater.cc:511] Master 127.11.229.62:42581 requested a full tablet report, sending...
W20260923 17:27:10.944979 25164 heartbeater.cc:644] Failed to heartbeat to 127.11.229.62:42581 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 17:27:10.945096 25164 heartbeater.cc:464] Registering TS with master...
I20260923 17:27:10.945289 25164 heartbeater.cc:511] Master 127.11.229.62:42581 requested a full tablet report, sending...
I20260923 17:27:10.946220 25164 heartbeater.cc:464] Registering TS with master...
I20260923 17:27:10.946377 25164 heartbeater.cc:511] Master 127.11.229.62:42581 requested a full tablet report, sending...
W20260923 17:27:10.947265 25164 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 17:27:11.050969 25166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:11.051316 25166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:11.051383 25166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:11.057475 25166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:11.057557 25166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:11.057652 25166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.2
I20260923 17:27:11.065203 25166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.229.20:60021/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.2:36279
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.229.2
--webserver_port=38867
--tserver_master_addrs=127.11.229.62:42581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.2
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:27:11.066570 25166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:11.067809 25166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:11.080138 25176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:11.080444 25166 server_base.cc:1034] running on GCE node
W20260923 17:27:11.080138 25173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:11.082594 25174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:11.083103 25166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:11.083823 25166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:11.085008 25166 hybrid_clock.cc:648] HybridClock initialized: now 1790184431084992 us; error 34 us; skew 500 ppm
Sep 23 17:27:11 dist-test-slave-ptl9 krb5kdc[23970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184431, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:11.088403 25166 init.cc:377] Logged in from keytab as kudu/127.11.229.2@KRBTEST.COM (short username kudu)
I20260923 17:27:11.089541 25166 webserver.cc:460] Webserver started at http://127.11.229.2:38867/ using document root <none> and password file <none>
I20260923 17:27:11.090255 25166 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:11.090335 25166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 17:27:11 dist-test-slave-ptl9 krb5kdc[23970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184431, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for HTTP/127.11.229.20@KRBTEST.COM
I20260923 17:27:11.109397 25166 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:27:11.111931 25184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:11.113540 25166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:27:11.113670 25166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-1/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-1/wal
uuid: "978632d31add4f05ad4a37d8b1883d51"
format_stamp: "Formatted at 2026-09-23 17:27:09 on dist-test-slave-ptl9"
server_key: "f41e73e4b9a002dc7ef1b2d563aab668"
server_key_iv: "ac19df5643d81242b8eb4c7cf614e8d2"
server_key_version: "kuduclusterkey@0"
I20260923 17:27:11.114287 25166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:11.129228 25166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:11.132738 25166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:11.132995 25166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:11.133786 25166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:27:11.134871 25166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:27:11.134977 25166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:11.135083 25166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:27:11.135174 25166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:11.146838 25166 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.2:36279
I20260923 17:27:11.146916 25297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.2:36279 every 8 connection(s)
I20260923 17:27:11.148262 25166 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
I20260923 17:27:11.156754 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 25166
I20260923 17:27:11.157351 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.229.20:60021/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.3:38155
--local_ip_for_outbound_sockets=127.11.229.3
--tserver_master_addrs=127.11.229.62:42581
--webserver_port=44173
--webserver_interface=127.11.229.3
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.229.20:60021/kms with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:27:11 dist-test-slave-ptl9 krb5kdc[23970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184431, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:27:11.167083 25168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:11.150630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:46563 (local address 127.11.229.62:42581)
0923 17:27:11.152285 (+ 1655us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:11.152290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:11.152309 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:27:11.153194 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:11.153197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:11.153295 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:11.153394 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:11.154907 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:11.155744 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:11.157536 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:11.157910 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:11.157993 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:11.161140 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:11.161144 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:11.161147 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:11.161177 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:11.163162 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:11.163810 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:11.163817 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:11.163822 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:11.163923 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:11.164365 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:11.164371 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:11.164375 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:11.164643 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:11.164761 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:11.166741 (+ 1980us) server_negotiation.cc:299] Negotiation successful
0923 17:27:11.166902 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1552}
I20260923 17:27:11.169117 25300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:11.151177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:42581 (local address 127.11.229.2:46563)
0923 17:27:11.151754 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:11.151815 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:11.152927 (+ 1112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:11.153611 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:11.153621 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:11.153988 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:11.154706 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:11.154737 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:11.155915 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:11.155919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:11.157387 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:11.157399 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:11.158212 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:11.158217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:11.158429 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:11.159114 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:11.159141 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:11.160931 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:11.163299 (+ 2368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:11.163305 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:11.163323 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:11.163681 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:11.164036 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:11.164039 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:11.164041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:11.164249 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:11.164778 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:11.164785 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:11.165123 (+ 338us) client_negotiation.cc:769] Sending connection context
0923 17:27:11.167587 (+ 2464us) client_negotiation.cc:240] Negotiation successful
0923 17:27:11.167948 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":141,"threads_started":1}
I20260923 17:27:11.170822 25298 heartbeater.cc:347] Connected to a master server at 127.11.229.62:42581
I20260923 17:27:11.171108 25298 heartbeater.cc:464] Registering TS with master...
I20260923 17:27:11.171749 25298 heartbeater.cc:511] Master 127.11.229.62:42581 requested a full tablet report, sending...
W20260923 17:27:11.173704 25298 heartbeater.cc:644] Failed to heartbeat to 127.11.229.62:42581 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 17:27:11.173823 25298 heartbeater.cc:464] Registering TS with master...
I20260923 17:27:11.174000 25298 heartbeater.cc:511] Master 127.11.229.62:42581 requested a full tablet report, sending...
I20260923 17:27:11.174872 25298 heartbeater.cc:464] Registering TS with master...
I20260923 17:27:11.175027 25298 heartbeater.cc:511] Master 127.11.229.62:42581 requested a full tablet report, sending...
W20260923 17:27:11.175752 25298 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 17:27:11.302564 25301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:11.302883 25301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:11.302955 25301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:11.307724 25301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:11.307818 25301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:11.307915 25301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.3
I20260923 17:27:11.313901 25301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.229.20:60021/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.3:38155
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.229.3
--webserver_port=44173
--tserver_master_addrs=127.11.229.62:42581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.3
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:27:11.315038 25301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:11.316157 25301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:11.326728 25307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:11.326745 25310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:11.327426 25301 server_base.cc:1034] running on GCE node
W20260923 17:27:11.327500 25308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:11.328078 25301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:11.328907 25301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:11.330143 25301 hybrid_clock.cc:648] HybridClock initialized: now 1790184431330119 us; error 40 us; skew 500 ppm
Sep 23 17:27:11 dist-test-slave-ptl9 krb5kdc[23970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184431, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:11.333767 25301 init.cc:377] Logged in from keytab as kudu/127.11.229.3@KRBTEST.COM (short username kudu)
I20260923 17:27:11.334887 25301 webserver.cc:460] Webserver started at http://127.11.229.3:44173/ using document root <none> and password file <none>
I20260923 17:27:11.335708 25301 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:11.335856 25301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 17:27:11 dist-test-slave-ptl9 krb5kdc[23970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184431, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for HTTP/127.11.229.20@KRBTEST.COM
I20260923 17:27:11.370771 25301 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.003s sys 0.000s
I20260923 17:27:11.374755 25318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:11.376543 25301 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260923 17:27:11.376704 25301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-2/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-2/wal
uuid: "0c88a5d0eb1d4fd2b580c79e883ef210"
format_stamp: "Formatted at 2026-09-23 17:27:10 on dist-test-slave-ptl9"
server_key: "b8d2db4ad8865567cd02329b322e7fde"
server_key_iv: "dffe4f6e61cf4eacd3d4e031b0b5ca9f"
server_key_version: "kuduclusterkey@0"
I20260923 17:27:11.377517 25301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:11.401110 25301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:11.404932 25301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:11.405170 25301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:11.405885 25301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:27:11.406952 25301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:27:11.407044 25301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:11.407122 25301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:27:11.407181 25301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:11.418185 25301 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.3:38155
I20260923 17:27:11.418358 25431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.3:38155 every 8 connection(s)
I20260923 17:27:11.419618 25301 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
I20260923 17:27:11.428543 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 25301
Sep 23 17:27:11 dist-test-slave-ptl9 krb5kdc[23970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184431, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:27:11.435115 25168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:11.422085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:58579 (local address 127.11.229.62:42581)
0923 17:27:11.422254 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:11.422259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:11.423152 (+ 893us) server_negotiation.cc:407] Connection header received
0923 17:27:11.424157 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:11.424161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:11.424226 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:11.424340 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:11.425633 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:11.426163 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:11.427311 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:11.427563 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:11.427690 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:11.430850 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:11.430856 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:11.430860 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:11.430890 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:11.432598 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:11.433249 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:11.433253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:11.433255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:11.433367 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:11.433759 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:11.433762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:11.433764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:11.433968 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:11.434040 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:11.434731 (+ 691us) server_negotiation.cc:299] Negotiation successful
0923 17:27:11.434878 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 17:27:11.437003 25434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:11.422506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:42581 (local address 127.11.229.3:58579)
0923 17:27:11.422982 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:11.423035 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:11.423942 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:11.424492 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:11.424499 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:11.424773 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:11.425477 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:11.425493 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:11.426290 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:11.426294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:11.427185 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:11.427194 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:11.427731 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:11.427735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:11.427909 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:11.428600 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:11.428625 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:11.430642 (+ 2017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:11.432725 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:11.432731 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:11.432747 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:11.433121 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:11.433483 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:11.433486 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:11.433488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:11.433655 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:11.434117 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:11.434137 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:11.434446 (+ 309us) client_negotiation.cc:769] Sending connection context
0923 17:27:11.435470 (+ 1024us) client_negotiation.cc:240] Negotiation successful
0923 17:27:11.435832 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":140,"threads_started":1}
I20260923 17:27:11.438401 25432 heartbeater.cc:347] Connected to a master server at 127.11.229.62:42581
I20260923 17:27:11.438684 25432 heartbeater.cc:464] Registering TS with master...
I20260923 17:27:11.439347 25432 heartbeater.cc:511] Master 127.11.229.62:42581 requested a full tablet report, sending...
W20260923 17:27:11.441008 25432 heartbeater.cc:644] Failed to heartbeat to 127.11.229.62:42581 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 17:27:11.441195 25432 heartbeater.cc:464] Registering TS with master...
I20260923 17:27:11.441453 25432 heartbeater.cc:511] Master 127.11.229.62:42581 requested a full tablet report, sending...
I20260923 17:27:11.442432 25432 heartbeater.cc:464] Registering TS with master...
I20260923 17:27:11.442652 25432 heartbeater.cc:511] Master 127.11.229.62:42581 requested a full tablet report, sending...
W20260923 17:27:11.443598 25432 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 17:27:11.444430 25435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:11.429309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42672 (local address 127.11.229.62:42581)
0923 17:27:11.430085 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:11.430089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:11.430101 (+ 12us) server_negotiation.cc:407] Connection header received
0923 17:27:11.430138 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:11.430140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:11.430186 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:11.430273 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:11.431956 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:11.432800 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:11.434044 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:11.434350 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:11.434448 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:11.436319 (+ 1871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:11.436323 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:11.436325 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:11.436349 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:11.438935 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:11.439455 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:11.439458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:11.439460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:11.439542 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:11.439864 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:11.439867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:11.439869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:11.443300 (+ 3431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:11.443505 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:11.444096 (+ 591us) server_negotiation.cc:299] Negotiation successful
0923 17:27:11.444247 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":86,"threads_started":1}
Sep 23 17:27:11 dist-test-slave-ptl9 krb5kdc[23970](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790184431, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:11.947495 25164 heartbeater.cc:464] Registering TS with master...
I20260923 17:27:11.947822 25164 heartbeater.cc:511] Master 127.11.229.62:42581 requested a full tablet report, sending...
I20260923 17:27:11.949045 24966 ts_manager.cc:194] Registered new tserver with Master: 04bb5ca847f84da29571b62429b9b56c (127.11.229.1:40761)
I20260923 17:27:11.950832 24966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.1@KRBTEST.COM'} at 127.11.229.1:59965
Sep 23 17:27:12 dist-test-slave-ptl9 krb5kdc[23970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184431, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for HTTP/127.11.229.20@KRBTEST.COM
I20260923 17:27:12.175981 25298 heartbeater.cc:464] Registering TS with master...
I20260923 17:27:12.176292 25298 heartbeater.cc:511] Master 127.11.229.62:42581 requested a full tablet report, sending...
I20260923 17:27:12.177089 24965 ts_manager.cc:194] Registered new tserver with Master: 978632d31add4f05ad4a37d8b1883d51 (127.11.229.2:36279)
I20260923 17:27:12.177950 24965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.2@KRBTEST.COM'} at 127.11.229.2:46563
Sep 23 17:27:12 dist-test-slave-ptl9 krb5kdc[23970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184431, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for HTTP/127.11.229.20@KRBTEST.COM
I20260923 17:27:12.443840 25432 heartbeater.cc:464] Registering TS with master...
I20260923 17:27:12.444150 25432 heartbeater.cc:511] Master 127.11.229.62:42581 requested a full tablet report, sending...
I20260923 17:27:12.444922 24965 ts_manager.cc:194] Registered new tserver with Master: 0c88a5d0eb1d4fd2b580c79e883ef210 (127.11.229.3:38155)
I20260923 17:27:12.445636 24965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.3@KRBTEST.COM'} at 127.11.229.3:58579
I20260923 17:27:12.453125 12180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:27:12.476136 25454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:12.455185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42680 (local address 127.11.229.62:42581)
0923 17:27:12.455687 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:12.455692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:12.456394 (+ 702us) server_negotiation.cc:407] Connection header received
0923 17:27:12.456600 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:12.456604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:12.456658 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:12.456737 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:12.457793 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:12.458600 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:12.459810 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:12.460159 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:12.460241 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:12.470557 (+ 10316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:12.470562 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:12.470565 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:12.470611 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:12.472811 (+ 2200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:12.474597 (+ 1786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:12.474601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:12.474603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:12.474725 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:12.475080 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:12.475083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:12.475085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:12.475327 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:12.475456 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:12.475816 (+ 360us) server_negotiation.cc:299] Negotiation successful
0923 17:27:12.475957 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":100,"threads_started":1}
I20260923 17:27:12.486370 24965 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:27:12.491039 24965 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:27:12.520732 25462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:12.510022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:40761 (local address 127.0.0.1:37994)
0923 17:27:12.512797 (+ 2775us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:12.512808 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:12.512871 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:12.514001 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:12.514007 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:12.514029 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:27:12.514849 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:12.514857 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:12.516263 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:12.516268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:12.518641 (+ 2373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:12.518653 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:12.520166 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:12.520170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:12.520285 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:12.520324 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 17:27:12.520435 (+ 111us) client_negotiation.cc:240] Negotiation successful
0923 17:27:12.520553 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2718,"thread_start_us":82,"threads_started":1}
I20260923 17:27:12.522349 25465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:12.510405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37994 (local address 127.11.229.1:40761)
0923 17:27:12.513362 (+ 2957us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:12.513370 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:12.513395 (+ 25us) server_negotiation.cc:407] Connection header received
0923 17:27:12.513471 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:12.513477 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:12.513698 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:12.513836 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:27:12.514986 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:12.516129 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:12.518787 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:12.519836 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:12.519943 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:12.520099 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:12.522097 (+ 1998us) server_negotiation.cc:299] Negotiation successful
0923 17:27:12.522205 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2827,"thread_start_us":95,"threads_started":1}
I20260923 17:27:12.525820 25091 tablet_service.cc:1467] Processing CreateTablet for tablet 9dbf54a968f645e5b2a6fe18ffc5445c (DEFAULT_TABLE table=test-table [id=0bc80c9cd8e14598b016f085cb5acc80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:27:12.527047 25091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dbf54a968f645e5b2a6fe18ffc5445c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:27:12.528766 25464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:12.511190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:38155 (local address 127.0.0.1:55298)
0923 17:27:12.513019 (+ 1829us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:12.513029 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:12.513111 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:12.519173 (+ 6062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:12.519176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:12.519195 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:27:12.519347 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:12.519353 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:12.524983 (+ 5630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:12.524987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:12.527258 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:12.527269 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:12.528340 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:12.528344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:12.528429 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:12.528455 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 17:27:12.528503 (+ 48us) client_negotiation.cc:240] Negotiation successful
0923 17:27:12.528575 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1782,"thread_start_us":95,"threads_started":1}
I20260923 17:27:12.529781 25467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:12.511377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55298 (local address 127.11.229.3:38155)
0923 17:27:12.518477 (+ 7100us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:12.518485 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:12.518551 (+ 66us) server_negotiation.cc:407] Connection header received
0923 17:27:12.518630 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:12.518636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:12.518857 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:12.519019 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:27:12.523497 (+ 4478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:12.524841 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:12.527416 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:12.528197 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:12.529340 (+ 1143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:12.529491 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:12.529544 (+ 53us) server_negotiation.cc:299] Negotiation successful
0923 17:27:12.529644 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6941,"thread_start_us":117,"threads_started":1}
I20260923 17:27:12.533088 25366 tablet_service.cc:1467] Processing CreateTablet for tablet 9dbf54a968f645e5b2a6fe18ffc5445c (DEFAULT_TABLE table=test-table [id=0bc80c9cd8e14598b016f085cb5acc80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:27:12.534264 25366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dbf54a968f645e5b2a6fe18ffc5445c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:27:12.536736 25466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:12.508127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:36279 (local address 127.0.0.1:57374)
0923 17:27:12.512394 (+ 4267us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:12.512445 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:12.512569 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:12.532434 (+ 19865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:12.532438 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:12.532458 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:27:12.532556 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:12.532562 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:12.534152 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:12.534155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:12.535261 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:12.535272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:12.536331 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:12.536334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:12.536435 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:12.536462 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 17:27:12.536508 (+ 46us) client_negotiation.cc:240] Negotiation successful
0923 17:27:12.536590 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4083,"thread_start_us":87,"threads_started":1}
I20260923 17:27:12.537452 25463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:12.508652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57374 (local address 127.11.229.2:36279)
0923 17:27:12.531942 (+ 23290us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:12.531950 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:12.531975 (+ 25us) server_negotiation.cc:407] Connection header received
0923 17:27:12.532053 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:12.532058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:12.532262 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:12.532732 (+ 470us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:27:12.532890 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:12.534012 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:12.535416 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:12.536187 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:12.536996 (+ 809us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:12.537134 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:12.537184 (+ 50us) server_negotiation.cc:299] Negotiation successful
0923 17:27:12.537294 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23168,"thread_start_us":77,"threads_started":1}
I20260923 17:27:12.540530 25232 tablet_service.cc:1467] Processing CreateTablet for tablet 9dbf54a968f645e5b2a6fe18ffc5445c (DEFAULT_TABLE table=test-table [id=0bc80c9cd8e14598b016f085cb5acc80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:27:12.541636 25232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dbf54a968f645e5b2a6fe18ffc5445c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:27:12.543622 25469 tablet_bootstrap.cc:492] T 9dbf54a968f645e5b2a6fe18ffc5445c P 0c88a5d0eb1d4fd2b580c79e883ef210: Bootstrap starting.
I20260923 17:27:12.546083 25469 tablet_bootstrap.cc:654] T 9dbf54a968f645e5b2a6fe18ffc5445c P 0c88a5d0eb1d4fd2b580c79e883ef210: Neither blocks nor log segments found. Creating new log.
I20260923 17:27:12.547013 25469 log.cc:824] T 9dbf54a968f645e5b2a6fe18ffc5445c P 0c88a5d0eb1d4fd2b580c79e883ef210: Log is configured to *not* fsync() on all Append() calls
I20260923 17:27:12.549587 25469 tablet_bootstrap.cc:492] T 9dbf54a968f645e5b2a6fe18ffc5445c P 0c88a5d0eb1d4fd2b580c79e883ef210: No bootstrap required, opened a new log
I20260923 17:27:12.549782 25469 ts_tablet_manager.cc:1397] T 9dbf54a968f645e5b2a6fe18ffc5445c P 0c88a5d0eb1d4fd2b580c79e883ef210: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.002s
I20260923 17:27:12.553668 25468 tablet_bootstrap.cc:492] T 9dbf54a968f645e5b2a6fe18ffc5445c P 04bb5ca847f84da29571b62429b9b56c: Bootstrap starting.
I20260923 17:27:12.554572 25469 raft_consensus.cc:348] T 9dbf54a968f645e5b2a6fe18ffc5445c P 0c88a5d0eb1d4fd2b580c79e883ef210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "978632d31add4f05ad4a37d8b1883d51" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36279 } } peers { permanent_uuid: "04bb5ca847f84da29571b62429b9b56c" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 40761 } } peers { permanent_uuid: "0c88a5d0eb1d4fd2b580c79e883ef210" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 38155 } }
I20260923 17:27:12.554965 25469 raft_consensus.cc:374] T 9dbf54a968f645e5b2a6fe18ffc5445c P 0c88a5d0eb1d4fd2b580c79e883ef210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:27:12.555097 25469 raft_consensus.cc:729] T 9dbf54a968f645e5b2a6fe18ffc5445c P 0c88a5d0eb1d4fd2b580c79e883ef210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c88a5d0eb1d4fd2b580c79e883ef210, State: Initialized, Role: FOLLOWER
I20260923 17:27:12.555687 25469 consensus_queue.cc:260] T 9dbf54a968f645e5b2a6fe18ffc5445c P 0c88a5d0eb1d4fd2b580c79e883ef210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "978632d31add4f05ad4a37d8b1883d51" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36279 } } peers { permanent_uuid: "04bb5ca847f84da29571b62429b9b56c" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 40761 } } peers { permanent_uuid: "0c88a5d0eb1d4fd2b580c79e883ef210" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 38155 } }
I20260923 17:27:12.556171 25468 tablet_bootstrap.cc:654] T 9dbf54a968f645e5b2a6fe18ffc5445c P 04bb5ca847f84da29571b62429b9b56c: Neither blocks nor log segments found. Creating new log.
I20260923 17:27:12.556957 25469 ts_tablet_manager.cc:1428] T 9dbf54a968f645e5b2a6fe18ffc5445c P 0c88a5d0eb1d4fd2b580c79e883ef210: Time spent starting tablet: real 0.007s user 0.000s sys 0.006s
I20260923 17:27:12.557013 25468 log.cc:824] T 9dbf54a968f645e5b2a6fe18ffc5445c P 04bb5ca847f84da29571b62429b9b56c: Log is configured to *not* fsync() on all Append() calls
I20260923 17:27:12.557500 25432 heartbeater.cc:503] Master 127.11.229.62:42581 was elected leader, sending a full tablet report...
I20260923 17:27:12.561499 25471 tablet_bootstrap.cc:492] T 9dbf54a968f645e5b2a6fe18ffc5445c P 978632d31add4f05ad4a37d8b1883d51: Bootstrap starting.
I20260923 17:27:12.566799 25471 tablet_bootstrap.cc:654] T 9dbf54a968f645e5b2a6fe18ffc5445c P 978632d31add4f05ad4a37d8b1883d51: Neither blocks nor log segments found. Creating new log.
I20260923 17:27:12.567387 25468 tablet_bootstrap.cc:492] T 9dbf54a968f645e5b2a6fe18ffc5445c P 04bb5ca847f84da29571b62429b9b56c: No bootstrap required, opened a new log
I20260923 17:27:12.567601 25468 ts_tablet_manager.cc:1397] T 9dbf54a968f645e5b2a6fe18ffc5445c P 04bb5ca847f84da29571b62429b9b56c: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20260923 17:27:12.567912 25471 log.cc:824] T 9dbf54a968f645e5b2a6fe18ffc5445c P 978632d31add4f05ad4a37d8b1883d51: Log is configured to *not* fsync() on all Append() calls
I20260923 17:27:12.571605 25468 raft_consensus.cc:348] T 9dbf54a968f645e5b2a6fe18ffc5445c P 04bb5ca847f84da29571b62429b9b56c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "978632d31add4f05ad4a37d8b1883d51" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36279 } } peers { permanent_uuid: "04bb5ca847f84da29571b62429b9b56c" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 40761 } } peers { permanent_uuid: "0c88a5d0eb1d4fd2b580c79e883ef210" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 38155 } }
I20260923 17:27:12.571902 25468 raft_consensus.cc:374] T 9dbf54a968f645e5b2a6fe18ffc5445c P 04bb5ca847f84da29571b62429b9b56c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:27:12.571951 25468 raft_consensus.cc:729] T 9dbf54a968f645e5b2a6fe18ffc5445c P 04bb5ca847f84da29571b62429b9b56c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04bb5ca847f84da29571b62429b9b56c, State: Initialized, Role: FOLLOWER
I20260923 17:27:12.572474 25468 consensus_queue.cc:260] T 9dbf54a968f645e5b2a6fe18ffc5445c P 04bb5ca847f84da29571b62429b9b56c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "978632d31add4f05ad4a37d8b1883d51" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36279 } } peers { permanent_uuid: "04bb5ca847f84da29571b62429b9b56c" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 40761 } } peers { permanent_uuid: "0c88a5d0eb1d4fd2b580c79e883ef210" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 38155 } }
I20260923 17:27:12.573230 25164 heartbeater.cc:503] Master 127.11.229.62:42581 was elected leader, sending a full tablet report...
I20260923 17:27:12.574777 25468 ts_tablet_manager.cc:1428] T 9dbf54a968f645e5b2a6fe18ffc5445c P 04bb5ca847f84da29571b62429b9b56c: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260923 17:27:12.579339 25471 tablet_bootstrap.cc:492] T 9dbf54a968f645e5b2a6fe18ffc5445c P 978632d31add4f05ad4a37d8b1883d51: No bootstrap required, opened a new log
I20260923 17:27:12.579629 25471 ts_tablet_manager.cc:1397] T 9dbf54a968f645e5b2a6fe18ffc5445c P 978632d31add4f05ad4a37d8b1883d51: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20260923 17:27:12.584160 25471 raft_consensus.cc:348] T 9dbf54a968f645e5b2a6fe18ffc5445c P 978632d31add4f05ad4a37d8b1883d51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "978632d31add4f05ad4a37d8b1883d51" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36279 } } peers { permanent_uuid: "04bb5ca847f84da29571b62429b9b56c" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 40761 } } peers { permanent_uuid: "0c88a5d0eb1d4fd2b580c79e883ef210" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 38155 } }
I20260923 17:27:12.584594 25471 raft_consensus.cc:374] T 9dbf54a968f645e5b2a6fe18ffc5445c P 978632d31add4f05ad4a37d8b1883d51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:27:12.584712 25471 raft_consensus.cc:729] T 9dbf54a968f645e5b2a6fe18ffc5445c P 978632d31add4f05ad4a37d8b1883d51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 978632d31add4f05ad4a37d8b1883d51, State: Initialized, Role: FOLLOWER
I20260923 17:27:12.585361 25471 consensus_queue.cc:260] T 9dbf54a968f645e5b2a6fe18ffc5445c P 978632d31add4f05ad4a37d8b1883d51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "978632d31add4f05ad4a37d8b1883d51" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36279 } } peers { permanent_uuid: "04bb5ca847f84da29571b62429b9b56c" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 40761 } } peers { permanent_uuid: "0c88a5d0eb1d4fd2b580c79e883ef210" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 38155 } }
I20260923 17:27:12.586612 25471 ts_tablet_manager.cc:1428] T 9dbf54a968f645e5b2a6fe18ffc5445c P 978632d31add4f05ad4a37d8b1883d51: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260923 17:27:12.587041 25298 heartbeater.cc:503] Master 127.11.229.62:42581 was elected leader, sending a full tablet report...
W20260923 17:27:12.651611 25299 tablet.cc:2369] T 9dbf54a968f645e5b2a6fe18ffc5445c P 978632d31add4f05ad4a37d8b1883d51: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:27:12.663159 25165 tablet.cc:2369] T 9dbf54a968f645e5b2a6fe18ffc5445c P 04bb5ca847f84da29571b62429b9b56c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:27:12.670080 25472 raft_consensus.cc:482] T 9dbf54a968f645e5b2a6fe18ffc5445c P 0c88a5d0eb1d4fd2b580c79e883ef210 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:27:12.670316 25472 raft_consensus.cc:504] T 9dbf54a968f645e5b2a6fe18ffc5445c P 0c88a5d0eb1d4fd2b580c79e883ef210 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "978632d31add4f05ad4a37d8b1883d51" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36279 } } peers { permanent_uuid: "04bb5ca847f84da29571b62429b9b56c" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 40761 } } peers { permanent_uuid: "0c88a5d0eb1d4fd2b580c79e883ef210" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 38155 } }
W20260923 17:27:12.671631 25433 tablet.cc:2369] T 9dbf54a968f645e5b2a6fe18ffc5445c P 0c88a5d0eb1d4fd2b580c79e883ef210: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:27:12.671720 25472 leader_election.cc:288] T 9dbf54a968f645e5b2a6fe18ffc5445c P 0c88a5d0eb1d4fd2b580c79e883ef210 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 978632d31add4f05ad4a37d8b1883d51 (127.11.229.2:36279), 04bb5ca847f84da29571b62429b9b56c (127.11.229.1:40761)
I20260923 17:27:12.677199 25463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:12.672739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:55687 (local address 127.11.229.2:36279)
0923 17:27:12.672877 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:12.672882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:12.672897 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:27:12.672960 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:12.672965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:12.673015 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:12.673105 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:27:12.673585 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:12.674380 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:12.675915 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:12.676485 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:12.676577 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:12.676630 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:12.676986 (+ 356us) server_negotiation.cc:299] Negotiation successful
0923 17:27:12.677065 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 17:27:12.677650 25477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:12.672128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:36279 (local address 127.11.229.3:55687)
0923 17:27:12.672459 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:12.672475 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:12.672585 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:12.673246 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:12.673250 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:12.673296 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:27:12.673444 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:12.673451 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:12.674527 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:12.674530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:12.675779 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:12.675790 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:12.676735 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:12.676739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:12.676843 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:12.676867 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 17:27:12.677436 (+ 569us) client_negotiation.cc:240] Negotiation successful
0923 17:27:12.677517 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":163,"threads_started":1}
I20260923 17:27:12.678298 25252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dbf54a968f645e5b2a6fe18ffc5445c" candidate_uuid: "0c88a5d0eb1d4fd2b580c79e883ef210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "978632d31add4f05ad4a37d8b1883d51" is_pre_election: true
I20260923 17:27:12.678638 25252 raft_consensus.cc:2454] T 9dbf54a968f645e5b2a6fe18ffc5445c P 978632d31add4f05ad4a37d8b1883d51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c88a5d0eb1d4fd2b580c79e883ef210 in term 0.
I20260923 17:27:12.679037 25475 raft_consensus.cc:482] T 9dbf54a968f645e5b2a6fe18ffc5445c P 04bb5ca847f84da29571b62429b9b56c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:27:12.679167 25319 leader_election.cc:302] T 9dbf54a968f645e5b2a6fe18ffc5445c P 0c88a5d0eb1d4fd2b580c79e883ef210 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c88a5d0eb1d4fd2b580c79e883ef210, 978632d31add4f05ad4a37d8b1883d51; no voters:
I20260923 17:27:12.679221 25475 raft_consensus.cc:504] T 9dbf54a968f645e5b2a6fe18ffc5445c P 04bb5ca847f84da29571b62429b9b56c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "978632d31add4f05ad4a37d8b1883d51" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36279 } } peers { permanent_uuid: "04bb5ca847f84da29571b62429b9b56c" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 40761 } } peers { permanent_uuid: "0c88a5d0eb1d4fd2b580c79e883ef210" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 38155 } }
I20260923 17:27:12.679471 25472 raft_consensus.cc:2781] T 9dbf54a968f645e5b2a6fe18ffc5445c P 0c88a5d0eb1d4fd2b580c79e883ef210 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:27:12.679539 25472 raft_consensus.cc:482] T 9dbf54a968f645e5b2a6fe18ffc5445c P 0c88a5d0eb1d4fd2b580c79e883ef210 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:27:12.679579 25472 raft_consensus.cc:3037] T 9dbf54a968f645e5b2a6fe18ffc5445c P 0c88a5d0eb1d4fd2b580c79e883ef210 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:27:12.680477 25475 leader_election.cc:288] T 9dbf54a968f645e5b2a6fe18ffc5445c P 04bb5ca847f84da29571b62429b9b56c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 978632d31add4f05ad4a37d8b1883d51 (127.11.229.2:36279), 0c88a5d0eb1d4fd2b580c79e883ef210 (127.11.229.3:38155)
I20260923 17:27:12.680951 25472 raft_consensus.cc:504] T 9dbf54a968f645e5b2a6fe18ffc5445c P 0c88a5d0eb1d4fd2b580c79e883ef210 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "978632d31add4f05ad4a37d8b1883d51" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36279 } } peers { permanent_uuid: "04bb5ca847f84da29571b62429b9b56c" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 40761 } } peers { permanent_uuid: "0c88a5d0eb1d4fd2b580c79e883ef210" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 38155 } }
I20260923 17:27:12.681365 25472 leader_election.cc:288] T 9dbf54a968f645e5b2a6fe18ffc5445c P 0c88a5d0eb1d4fd2b580c79e883ef210 [CANDIDATE]: Term 1 election: Requested vote from peers 978632d31add4f05ad4a37d8b1883d51 (127.11.229.2:36279), 04bb5ca847f84da29571b62429b9b56c (127.11.229.1:40761)
I20260923 17:27:12.682093 25252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dbf54a968f645e5b2a6fe18ffc5445c" candidate_uuid: "0c88a5d0eb1d4fd2b580c79e883ef210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "978632d31add4f05ad4a37d8b1883d51"
I20260923 17:27:12.682220 25252 raft_consensus.cc:3037] T 9dbf54a968f645e5b2a6fe18ffc5445c P 978632d31add4f05ad4a37d8b1883d51 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:27:12.683526 25252 raft_consensus.cc:2454] T 9dbf54a968f645e5b2a6fe18ffc5445c P 978632d31add4f05ad4a37d8b1883d51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c88a5d0eb1d4fd2b580c79e883ef210 in term 1.
I20260923 17:27:12.683861 25319 leader_election.cc:302] T 9dbf54a968f645e5b2a6fe18ffc5445c P 0c88a5d0eb1d4fd2b580c79e883ef210 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c88a5d0eb1d4fd2b580c79e883ef210, 978632d31add4f05ad4a37d8b1883d51; no voters:
I20260923 17:27:12.684026 25472 raft_consensus.cc:2781] T 9dbf54a968f645e5b2a6fe18ffc5445c P 0c88a5d0eb1d4fd2b580c79e883ef210 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:27:12.684224 25472 raft_consensus.cc:686] T 9dbf54a968f645e5b2a6fe18ffc5445c P 0c88a5d0eb1d4fd2b580c79e883ef210 [term 1 LEADER]: Becoming Leader. State: Replica: 0c88a5d0eb1d4fd2b580c79e883ef210, State: Running, Role: LEADER
I20260923 17:27:12.684576 25472 consensus_queue.cc:237] T 9dbf54a968f645e5b2a6fe18ffc5445c P 0c88a5d0eb1d4fd2b580c79e883ef210 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "978632d31add4f05ad4a37d8b1883d51" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36279 } } peers { permanent_uuid: "04bb5ca847f84da29571b62429b9b56c" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 40761 } } peers { permanent_uuid: "0c88a5d0eb1d4fd2b580c79e883ef210" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 38155 } }
I20260923 17:27:12.688328 24965 catalog_manager.cc:5495] T 9dbf54a968f645e5b2a6fe18ffc5445c P 0c88a5d0eb1d4fd2b580c79e883ef210 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c88a5d0eb1d4fd2b580c79e883ef210 (127.11.229.3). New cstate: current_term: 1 leader_uuid: "0c88a5d0eb1d4fd2b580c79e883ef210" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "978632d31add4f05ad4a37d8b1883d51" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04bb5ca847f84da29571b62429b9b56c" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 40761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c88a5d0eb1d4fd2b580c79e883ef210" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 38155 } health_report { overall_health: HEALTHY } } }
I20260923 17:27:12.695508 25465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:12.673450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:51485 (local address 127.11.229.1:40761)
0923 17:27:12.673595 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:12.673599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:12.690441 (+ 16842us) server_negotiation.cc:407] Connection header received
0923 17:27:12.690660 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:12.690664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:12.690733 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:12.690780 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:27:12.691183 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:12.691930 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:12.693748 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:12.694279 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:12.694366 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:12.694419 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:12.695293 (+ 874us) server_negotiation.cc:299] Negotiation successful
0923 17:27:12.695373 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45,"spinlock_wait_cycles":384}
I20260923 17:27:12.695955 25477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:12.673025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:40761 (local address 127.11.229.3:51485)
0923 17:27:12.690362 (+ 17337us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:12.690379 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:12.690514 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:12.690954 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:12.690957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:12.690975 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:27:12.691063 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:12.691071 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:12.692421 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:12.692424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:12.693593 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:12.693604 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:12.695058 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:12.695061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:12.695149 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:12.695174 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 17:27:12.695734 (+ 560us) client_negotiation.cc:240] Negotiation successful
0923 17:27:12.695825 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8960,"thread_start_us":98,"threads_started":1}
I20260923 17:27:12.696609 25118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dbf54a968f645e5b2a6fe18ffc5445c" candidate_uuid: "0c88a5d0eb1d4fd2b580c79e883ef210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04bb5ca847f84da29571b62429b9b56c" is_pre_election: true
I20260923 17:27:12.696861 25118 raft_consensus.cc:2454] T 9dbf54a968f645e5b2a6fe18ffc5445c P 04bb5ca847f84da29571b62429b9b56c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c88a5d0eb1d4fd2b580c79e883ef210 in term 0.
I20260923 17:27:12.697213 25117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dbf54a968f645e5b2a6fe18ffc5445c" candidate_uuid: "0c88a5d0eb1d4fd2b580c79e883ef210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04bb5ca847f84da29571b62429b9b56c"
I20260923 17:27:12.697355 25117 raft_consensus.cc:3037] T 9dbf54a968f645e5b2a6fe18ffc5445c P 04bb5ca847f84da29571b62429b9b56c [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:27:12.697428 25463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:12.681855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:45429 (local address 127.11.229.2:36279)
0923 17:27:12.681955 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:12.681958 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:12.687322 (+ 5364us) server_negotiation.cc:407] Connection header received
0923 17:27:12.687535 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:12.687539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:12.687586 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:12.687631 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:27:12.688079 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:12.688843 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:12.695827 (+ 6984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:12.696429 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:12.696529 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:12.696582 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:12.697220 (+ 638us) server_negotiation.cc:299] Negotiation successful
0923 17:27:12.697316 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260923 17:27:12.698045 25480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:12.680798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:36279 (local address 127.11.229.1:45429)
0923 17:27:12.687226 (+ 6428us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:12.687243 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:12.687408 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:12.687778 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:12.687780 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:12.687799 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:27:12.687944 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:12.687951 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:12.689146 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:12.689149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:12.690379 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:12.690393 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:12.696883 (+ 6490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:12.696886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:12.697045 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:12.697075 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 17:27:12.697769 (+ 694us) client_negotiation.cc:240] Negotiation successful
0923 17:27:12.697861 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6334,"thread_start_us":111,"threads_started":1}
I20260923 17:27:12.698750 25117 raft_consensus.cc:2454] T 9dbf54a968f645e5b2a6fe18ffc5445c P 04bb5ca847f84da29571b62429b9b56c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c88a5d0eb1d4fd2b580c79e883ef210 in term 1.
I20260923 17:27:12.698772 25252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dbf54a968f645e5b2a6fe18ffc5445c" candidate_uuid: "04bb5ca847f84da29571b62429b9b56c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "978632d31add4f05ad4a37d8b1883d51" is_pre_election: true
I20260923 17:27:12.698973 25252 raft_consensus.cc:2379] T 9dbf54a968f645e5b2a6fe18ffc5445c P 978632d31add4f05ad4a37d8b1883d51 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 04bb5ca847f84da29571b62429b9b56c in current term 1: Already voted for candidate 0c88a5d0eb1d4fd2b580c79e883ef210 in this term.
I20260923 17:27:12.700134 25467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:12.681567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:43175 (local address 127.11.229.3:38155)
0923 17:27:12.681697 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:12.681701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:12.690739 (+ 9038us) server_negotiation.cc:407] Connection header received
0923 17:27:12.690954 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:12.690957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:12.691034 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:12.691080 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:27:12.691495 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:12.692178 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:12.698808 (+ 6630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:12.699390 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:12.699476 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:12.699520 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:12.699947 (+ 427us) server_negotiation.cc:299] Negotiation successful
0923 17:27:12.700022 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37,"spinlock_wait_cycles":14208}
I20260923 17:27:12.700606 25479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:12.681176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:38155 (local address 127.11.229.1:43175)
0923 17:27:12.690664 (+ 9488us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:12.690677 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:12.690841 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:12.691210 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:12.691215 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:12.691233 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:27:12.691335 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:12.691340 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:12.692319 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:12.692322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:12.693593 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:12.693604 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:12.699693 (+ 6089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:12.699696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:12.699803 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:12.699827 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 17:27:12.700414 (+ 587us) client_negotiation.cc:240] Negotiation successful
0923 17:27:12.700503 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9403,"spinlock_wait_cycles":41216,"thread_start_us":104,"threads_started":1}
I20260923 17:27:12.701290 25386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dbf54a968f645e5b2a6fe18ffc5445c" candidate_uuid: "04bb5ca847f84da29571b62429b9b56c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c88a5d0eb1d4fd2b580c79e883ef210" is_pre_election: true
I20260923 17:27:12.702419 25051 leader_election.cc:302] T 9dbf54a968f645e5b2a6fe18ffc5445c P 04bb5ca847f84da29571b62429b9b56c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 04bb5ca847f84da29571b62429b9b56c; no voters: 0c88a5d0eb1d4fd2b580c79e883ef210, 978632d31add4f05ad4a37d8b1883d51
I20260923 17:27:12.702874 25475 raft_consensus.cc:2726] T 9dbf54a968f645e5b2a6fe18ffc5445c P 04bb5ca847f84da29571b62429b9b56c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260923 17:27:12.712659 25467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:12.708065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55300 (local address 127.11.229.3:38155)
0923 17:27:12.708168 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:12.708172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:12.708185 (+ 13us) server_negotiation.cc:407] Connection header received
0923 17:27:12.708259 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:12.708262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:12.708320 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:12.708418 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:27:12.708770 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:12.709512 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:12.710679 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:12.710976 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:12.711066 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:12.711437 (+ 371us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:27:12.711957 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:27:12.712066 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:12.712461 (+ 395us) server_negotiation.cc:299] Negotiation successful
0923 17:27:12.712510 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 17:27:12.726307 25117 raft_consensus.cc:1261] T 9dbf54a968f645e5b2a6fe18ffc5445c P 04bb5ca847f84da29571b62429b9b56c [term 1 FOLLOWER]: Refusing update from remote peer 0c88a5d0eb1d4fd2b580c79e883ef210: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:27:12.726294 25252 raft_consensus.cc:1261] T 9dbf54a968f645e5b2a6fe18ffc5445c P 978632d31add4f05ad4a37d8b1883d51 [term 1 FOLLOWER]: Refusing update from remote peer 0c88a5d0eb1d4fd2b580c79e883ef210: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:27:12.727011 25472 consensus_queue.cc:1035] T 9dbf54a968f645e5b2a6fe18ffc5445c P 0c88a5d0eb1d4fd2b580c79e883ef210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04bb5ca847f84da29571b62429b9b56c" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 40761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:27:12.727521 25472 consensus_queue.cc:1035] T 9dbf54a968f645e5b2a6fe18ffc5445c P 0c88a5d0eb1d4fd2b580c79e883ef210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "978632d31add4f05ad4a37d8b1883d51" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 36279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:27:12.746659 25486 mvcc.cc:205] Tried to move back new op lower bound from 7332595436438601728 to 7332595436279406592. Current Snapshot: MvccSnapshot[applied={T|T < 7332595436438601728}]
I20260923 17:27:12.746924 25484 mvcc.cc:205] Tried to move back new op lower bound from 7332595436438601728 to 7332595436279406592. Current Snapshot: MvccSnapshot[applied={T|T < 7332595436438601728}]
I20260923 17:27:12.751886 25487 mvcc.cc:205] Tried to move back new op lower bound from 7332595436438601728 to 7332595436279406592. Current Snapshot: MvccSnapshot[applied={T|T < 7332595436438601728 or (T in {7332595436438601728})}]
I20260923 17:27:12.763850 24966 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:27:12.764048 24966 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:27:12.766913 24966 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a5b00a58242c45f6b707c53b9b06ad0f: Sending DeleteTablet for 3 replicas of tablet 9dbf54a968f645e5b2a6fe18ffc5445c
I20260923 17:27:12.767802 25366 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dbf54a968f645e5b2a6fe18ffc5445c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:27:12 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:55298
I20260923 17:27:12.767799 25091 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dbf54a968f645e5b2a6fe18ffc5445c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:27:12 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:37994
I20260923 17:27:12.767900 25232 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dbf54a968f645e5b2a6fe18ffc5445c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:27:12 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:57374
I20260923 17:27:12.768750 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 25004
I20260923 17:27:12.771770 25501 tablet_replica.cc:330] T 9dbf54a968f645e5b2a6fe18ffc5445c P 0c88a5d0eb1d4fd2b580c79e883ef210: stopping tablet replica
I20260923 17:27:12.772142 25501 raft_consensus.cc:2229] T 9dbf54a968f645e5b2a6fe18ffc5445c P 0c88a5d0eb1d4fd2b580c79e883ef210 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:27:12.772584 25501 raft_consensus.cc:2258] T 9dbf54a968f645e5b2a6fe18ffc5445c P 0c88a5d0eb1d4fd2b580c79e883ef210 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:27:12.773495 25502 tablet_replica.cc:330] T 9dbf54a968f645e5b2a6fe18ffc5445c P 978632d31add4f05ad4a37d8b1883d51: stopping tablet replica
I20260923 17:27:12.773890 25502 raft_consensus.cc:2229] T 9dbf54a968f645e5b2a6fe18ffc5445c P 978632d31add4f05ad4a37d8b1883d51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:27:12.774266 25502 raft_consensus.cc:2258] T 9dbf54a968f645e5b2a6fe18ffc5445c P 978632d31add4f05ad4a37d8b1883d51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:27:12.775874 25502 ts_tablet_manager.cc:1905] T 9dbf54a968f645e5b2a6fe18ffc5445c P 978632d31add4f05ad4a37d8b1883d51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:27:12.778616 25501 ts_tablet_manager.cc:1905] T 9dbf54a968f645e5b2a6fe18ffc5445c P 0c88a5d0eb1d4fd2b580c79e883ef210: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:27:12.778877 25502 ts_tablet_manager.cc:1918] T 9dbf54a968f645e5b2a6fe18ffc5445c P 978632d31add4f05ad4a37d8b1883d51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:27:12.779030 25502 log.cc:1196] T 9dbf54a968f645e5b2a6fe18ffc5445c P 978632d31add4f05ad4a37d8b1883d51: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-1/wal/wals/9dbf54a968f645e5b2a6fe18ffc5445c
I20260923 17:27:12.779418 25502 ts_tablet_manager.cc:1939] T 9dbf54a968f645e5b2a6fe18ffc5445c P 978632d31add4f05ad4a37d8b1883d51: Deleting consensus metadata
I20260923 17:27:12.780831 24950 catalog_manager.cc:4841] TS 978632d31add4f05ad4a37d8b1883d51 (127.11.229.2:36279): tablet 9dbf54a968f645e5b2a6fe18ffc5445c (table test-table [id=0bc80c9cd8e14598b016f085cb5acc80]) successfully deleted
I20260923 17:27:12.781435 25501 ts_tablet_manager.cc:1918] T 9dbf54a968f645e5b2a6fe18ffc5445c P 0c88a5d0eb1d4fd2b580c79e883ef210: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:27:12.781529 25501 log.cc:1196] T 9dbf54a968f645e5b2a6fe18ffc5445c P 0c88a5d0eb1d4fd2b580c79e883ef210: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184271597484-12180-0/minicluster-data/ts-2/wal/wals/9dbf54a968f645e5b2a6fe18ffc5445c
I20260923 17:27:12.781848 25501 ts_tablet_manager.cc:1939] T 9dbf54a968f645e5b2a6fe18ffc5445c P 0c88a5d0eb1d4fd2b580c79e883ef210: Deleting consensus metadata
I20260923 17:27:12.783159 24950 catalog_manager.cc:4841] TS 0c88a5d0eb1d4fd2b580c79e883ef210 (127.11.229.3:38155): tablet 9dbf54a968f645e5b2a6fe18ffc5445c (table test-table [id=0bc80c9cd8e14598b016f085cb5acc80]) successfully deleted
W20260923 17:27:12.783540 24953 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.229.1:40761 (error 108)
I20260923 17:27:12.783654 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 25166
I20260923 17:27:12.790306 25466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:12.789946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:40761 (local address 127.0.0.1:38006)
0923 17:27:12.790141 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:12.790213 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.1:40761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260923 17:27:12.790488 24953 catalog_manager.cc:4584] TS 04bb5ca847f84da29571b62429b9b56c (127.11.229.1:40761): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9dbf54a968f645e5b2a6fe18ffc5445c: Network error: Client connection negotiation failed: client connection to 127.11.229.1:40761: connect: Connection refused (error 111)
I20260923 17:27:12.792354 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 25301
I20260923 17:27:12.798151 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 24933
I20260923 17:27:12.807523 12180 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 17:27:13.142446 12180 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 17:27:13.142550 12180 mini_ranger.cc:66] Stopping Ranger...
I20260923 17:27:13.487524 12180 mini_ranger.cc:68] Stopped Ranger
2026-09-23 17:27:13.487 UTC [24000] LOG: received smart shutdown request
2026-09-23 17:27:13.492 UTC [24000] LOG: background worker "logical replication launcher" (PID 24009) exited with exit code 1
2026-09-23 17:27:13.492 UTC [24004] LOG: shutting down
2026-09-23 17:27:13.664 UTC [24000] LOG: database system is shut down
2026-09-23T17:27:13Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49753 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:27:13 dist-test-slave-ptl9 krb5kdc[25512](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:27:13 dist-test-slave-ptl9 krb5kdc[25512](info): set up 2 sockets
Sep 23 17:27:13 dist-test-slave-ptl9 krb5kdc[25512](info): commencing operation
krb5kdc: starting...
W20260923 17:27:14.841609 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:27:14 dist-test-slave-ptl9 krb5kdc[25512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:27:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:27:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.229.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
I20260923 17:27:14.964915 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:43351
--webserver_interface=127.11.229.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:43247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:43351,127.11.229.61:45601,127.11.229.60:45979
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:27:15.080205 25525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:15.080580 25525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:15.080681 25525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:15.084841 25525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:27:15.084960 25525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:15.085001 25525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:15.085063 25525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:27:15.085108 25525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:27:15.090641 25525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:43247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:43351,127.11.229.61:45601,127.11.229.60:45979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:43351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:27:15.091923 25525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:15.093135 25525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:15.102943 25530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:15.102947 25533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:15.103040 25525 server_base.cc:1034] running on GCE node
W20260923 17:27:15.102943 25531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:15.104024 25525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:15.105394 25525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:15.106588 25525 hybrid_clock.cc:648] HybridClock initialized: now 1790184435106567 us; error 44 us; skew 500 ppm
Sep 23 17:27:15 dist-test-slave-ptl9 krb5kdc[25512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184435, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:15.109642 25525 init.cc:377] Logged in from keytab as kudu/127.11.229.62@KRBTEST.COM (short username kudu)
I20260923 17:27:15.110596 25525 webserver.cc:460] Webserver started at http://127.11.229.62:46369/ using document root <none> and password file <none>
I20260923 17:27:15.111260 25525 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:15.111323 25525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:15.111557 25525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:27:15.113807 25525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "fa4b4079ec574eaca7d0b8b679017021"
format_stamp: "Formatted at 2026-09-23 17:27:15 on dist-test-slave-ptl9"
server_key: "713f98de97e7faaf4f5535860c77d9a2"
server_key_iv: "f6261a5947b4eeb7997f0e5873c50f6c"
server_key_version: "encryptionkey@0"
I20260923 17:27:15.114411 25525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "fa4b4079ec574eaca7d0b8b679017021"
format_stamp: "Formatted at 2026-09-23 17:27:15 on dist-test-slave-ptl9"
server_key: "713f98de97e7faaf4f5535860c77d9a2"
server_key_iv: "f6261a5947b4eeb7997f0e5873c50f6c"
server_key_version: "encryptionkey@0"
I20260923 17:27:15.118577 25525 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260923 17:27:15.121747 25540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:15.123095 25525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:27:15.123255 25525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "fa4b4079ec574eaca7d0b8b679017021"
format_stamp: "Formatted at 2026-09-23 17:27:15 on dist-test-slave-ptl9"
server_key: "713f98de97e7faaf4f5535860c77d9a2"
server_key_iv: "f6261a5947b4eeb7997f0e5873c50f6c"
server_key_version: "encryptionkey@0"
I20260923 17:27:15.123389 25525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:15.151993 25525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:15.155784 25525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:15.156023 25525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:15.163345 25525 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:43351
I20260923 17:27:15.163371 25592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:43351 every 8 connection(s)
I20260923 17:27:15.164319 25525 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:27:15.167164 25593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:27:15.170987 25593 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.229.62" port: 43351 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:27:15.172083 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 25525
I20260923 17:27:15.172219 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:27:15.172549 12180 external_mini_cluster.cc:1428] Setting key 5b15b2f4bdcdd085657f1fac265df388
WARNING: no policy specified for kudu/127.11.229.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.61@KRBTEST.COM" created.
Sep 23 17:27:15 dist-test-slave-ptl9 krb5kdc[25512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184435, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:27:15.190709 25596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.173516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:43351 (local address 127.0.0.1:51926)
0923 17:27:15.174096 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:15.174131 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:15.175374 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:15.176229 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:15.176234 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:15.176559 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:15.176815 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:15.176829 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.178703 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.178706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:15.179444 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:15.179454 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.180371 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.180373 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:15.180462 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:15.181096 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:15.181113 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:15.185138 (+ 4025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:15.187600 (+ 2462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:15.187612 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:15.187627 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:15.187960 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:15.188294 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:15.188300 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:15.188305 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:15.188469 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:15.189215 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:15.189221 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:15.189407 (+ 186us) client_negotiation.cc:769] Sending connection context
0923 17:27:15.189565 (+ 158us) client_negotiation.cc:240] Negotiation successful
0923 17:27:15.189858 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"spinlock_wait_cycles":7552,"thread_start_us":176,"threads_started":1}
I20260923 17:27:15.190716 25595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.173515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51926 (local address 127.11.229.62:43351)
0923 17:27:15.173953 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:15.173972 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:15.174245 (+ 273us) server_negotiation.cc:407] Connection header received
0923 17:27:15.175758 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:15.175766 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:15.176000 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:15.176205 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:15.176979 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.178570 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.179569 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.180214 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.180392 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:15.185357 (+ 4965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:15.185363 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:15.185376 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:15.185398 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:15.187473 (+ 2075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:15.188082 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:15.188087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:15.188088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:15.188194 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:15.188588 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:15.188596 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:15.188597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:15.189054 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:15.189204 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:15.189696 (+ 492us) server_negotiation.cc:299] Negotiation successful
0923 17:27:15.189974 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":281,"thread_start_us":132,"threads_started":1}
I20260923 17:27:15.193405 25593 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.229.61" port: 45601 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:27:15.194254 25596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.193939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:45601 (local address 127.0.0.1:48516)
0923 17:27:15.194072 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:15.194179 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.61:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 17:27:15.194480 25542 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.229.61:45601: connect: Connection refused (error 111)
I20260923 17:27:15.195343 25596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.195131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:45601 (local address 127.0.0.1:48520)
0923 17:27:15.195246 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:15.195289 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.61:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260923 17:27:15.195564 25593 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.229.61:45601: Network error: Client connection negotiation failed: client connection to 127.11.229.61:45601: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.61.keytab.
Entry for principal kudu/127.11.229.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.61.keytab.
I20260923 17:27:15.202487 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.61:45601
--webserver_interface=127.11.229.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.61.keytab
--principal=kudu/127.11.229.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:43247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:43351,127.11.229.61:45601,127.11.229.60:45979
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:27:15.244887 25593 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.229.61" port: 45601 } attempt: 1
I20260923 17:27:15.246011 25596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.245579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:45601 (local address 127.0.0.1:48532)
0923 17:27:15.245799 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:15.245889 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.61:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260923 17:27:15.246970 25596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.246708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:45601 (local address 127.0.0.1:48536)
0923 17:27:15.246831 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:15.246890 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.61:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260923 17:27:15.247123 25593 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.229.61:45601: Network error: Client connection negotiation failed: client connection to 127.11.229.61:45601: connect: Connection refused (error 111)
I20260923 17:27:15.315312 25593 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.229.61" port: 45601 } attempt: 2
I20260923 17:27:15.316252 25596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.315894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:45601 (local address 127.0.0.1:48552)
0923 17:27:15.316080 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:15.316155 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.61:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260923 17:27:15.317003 25596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.316778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:45601 (local address 127.0.0.1:48556)
0923 17:27:15.316892 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:15.316943 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.61:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260923 17:27:15.317159 25593 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.229.61:45601: Network error: Client connection negotiation failed: client connection to 127.11.229.61:45601: connect: Connection refused (error 111)
W20260923 17:27:15.318557 25600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:15.318811 25600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:15.318853 25600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:15.322922 25600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:27:15.322978 25600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:15.322999 25600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:15.323015 25600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:27:15.323031 25600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:27:15.328519 25600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:43247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:43351,127.11.229.61:45601,127.11.229.60:45979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.61.keytab
--principal=kudu/127.11.229.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.61:45601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.229.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:27:15.329666 25600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:15.330669 25600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:15.339318 25606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:15.339406 25608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:15.339318 25605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:15.339572 25600 server_base.cc:1034] running on GCE node
I20260923 17:27:15.340314 25600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:15.341434 25600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:15.342623 25600 hybrid_clock.cc:648] HybridClock initialized: now 1790184435342607 us; error 27 us; skew 500 ppm
Sep 23 17:27:15 dist-test-slave-ptl9 krb5kdc[25512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184435, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:15.345604 25600 init.cc:377] Logged in from keytab as kudu/127.11.229.61@KRBTEST.COM (short username kudu)
I20260923 17:27:15.346513 25600 webserver.cc:460] Webserver started at http://127.11.229.61:36155/ using document root <none> and password file <none>
I20260923 17:27:15.347126 25600 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:15.347224 25600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:15.347461 25600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:27:15.349617 25600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-1/data/instance:
uuid: "a8a9ae00dc0149fda4b630578d134ae1"
format_stamp: "Formatted at 2026-09-23 17:27:15 on dist-test-slave-ptl9"
server_key: "5575d15d38f661e7d03536c657a294b1"
server_key_iv: "638d208f417e40a343666ca1f19bbfe7"
server_key_version: "encryptionkey@0"
I20260923 17:27:15.350186 25600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-1/wal/instance:
uuid: "a8a9ae00dc0149fda4b630578d134ae1"
format_stamp: "Formatted at 2026-09-23 17:27:15 on dist-test-slave-ptl9"
server_key: "5575d15d38f661e7d03536c657a294b1"
server_key_iv: "638d208f417e40a343666ca1f19bbfe7"
server_key_version: "encryptionkey@0"
I20260923 17:27:15.354071 25600 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:27:15.356633 25615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:15.358108 25600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:27:15.358283 25600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-1/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-1/wal
uuid: "a8a9ae00dc0149fda4b630578d134ae1"
format_stamp: "Formatted at 2026-09-23 17:27:15 on dist-test-slave-ptl9"
server_key: "5575d15d38f661e7d03536c657a294b1"
server_key_iv: "638d208f417e40a343666ca1f19bbfe7"
server_key_version: "encryptionkey@0"
I20260923 17:27:15.358410 25600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:15.382581 25600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:15.386013 25600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:15.386233 25600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:15.393391 25600 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.61:45601
I20260923 17:27:15.393409 25667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.61:45601 every 8 connection(s)
I20260923 17:27:15.394507 25600 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-1/data/info.pb
I20260923 17:27:15.397498 25668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:27:15.399161 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 25600
I20260923 17:27:15.399282 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-1/wal/instance
I20260923 17:27:15.399616 12180 external_mini_cluster.cc:1428] Setting key 7f5ffb7712dc4bcdfa1f1cec7d88be9b
I20260923 17:27:15.402525 25668 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.229.62" port: 43351 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.11.229.60@KRBTEST.COM; defaulting to no policy
I20260923 17:27:15.408324 25593 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.229.61" port: 45601 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.60@KRBTEST.COM" created.
Sep 23 17:27:15 dist-test-slave-ptl9 krb5kdc[25512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184435, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.61@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
Sep 23 17:27:15 dist-test-slave-ptl9 krb5kdc[25512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184435, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for kudu/127.11.229.61@KRBTEST.COM
I20260923 17:27:15.419329 25596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.408849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:45601 (local address 127.0.0.1:48568)
0923 17:27:15.409023 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:15.409035 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:15.409126 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:15.409776 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:15.409779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:15.409977 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:15.410102 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:15.410109 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.411539 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.411542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:15.412119 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:15.412131 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.412829 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.412833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:15.412943 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:15.413477 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:15.413497 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:15.414997 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:15.417310 (+ 2313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:15.417314 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:15.417317 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:15.417538 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:15.417924 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:15.417927 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:15.417929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:15.418045 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:15.418827 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:15.418830 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:15.418921 (+ 91us) client_negotiation.cc:769] Sending connection context
0923 17:27:15.419024 (+ 103us) client_negotiation.cc:240] Negotiation successful
0923 17:27:15.419159 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260923 17:27:15.419996 25595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.404615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51930 (local address 127.11.229.62:43351)
0923 17:27:15.404756 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:15.404760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:15.406094 (+ 1334us) server_negotiation.cc:407] Connection header received
0923 17:27:15.407066 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:15.407069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:15.407138 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:15.407237 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:15.408675 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.409495 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.410906 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.411288 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.411380 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:15.414408 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:15.414413 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:15.414417 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:15.414446 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:15.416844 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:15.417548 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:15.417562 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:15.417567 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:15.417652 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:15.418060 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:15.418066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:15.418071 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:15.418268 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:15.418374 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:15.419770 (+ 1396us) server_negotiation.cc:299] Negotiation successful
0923 17:27:15.419888 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 17:27:15.420393 25671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.405181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:43351 (local address 127.0.0.1:51930)
0923 17:27:15.405790 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:15.405837 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:15.406833 (+ 996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:15.407422 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:15.407429 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:15.407786 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:15.408456 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:15.408478 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.409635 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.409641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:15.410749 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:15.410780 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.411469 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.411477 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:15.411748 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:15.412476 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:15.412493 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:15.414254 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:15.416971 (+ 2717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:15.416983 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:15.416998 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:15.417437 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:15.417761 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:15.417767 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:15.417771 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:15.417937 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:15.418390 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:15.418399 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:15.418655 (+ 256us) client_negotiation.cc:769] Sending connection context
0923 17:27:15.418830 (+ 175us) client_negotiation.cc:240] Negotiation successful
0923 17:27:15.419173 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":12,"thread_start_us":170,"threads_started":1}
I20260923 17:27:15.420836 25672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.408944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48568 (local address 127.11.229.61:45601)
0923 17:27:15.409359 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:15.409369 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:15.409390 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:27:15.409424 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:15.409428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:15.409595 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:15.409833 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:15.410217 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.411448 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.412257 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.412700 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.412783 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:15.415155 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:15.415165 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:15.415182 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:15.415213 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:15.417180 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:15.417669 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:15.417678 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:15.417682 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:15.417825 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:15.418143 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:15.418148 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:15.418152 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:15.418703 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:15.418853 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:15.419138 (+ 285us) server_negotiation.cc:299] Negotiation successful
0923 17:27:15.419302 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":97,"server-negotiator.queue_time_us":266,"thread_start_us":111,"threads_started":1}
I20260923 17:27:15.422353 25668 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.229.61" port: 45601 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:27:15.422787 25593 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45979 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:27:15.423563 25596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.423316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:45979 (local address 127.0.0.1:46462)
0923 17:27:15.423424 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:15.423488 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.60:45979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260923 17:27:15.424301 25596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.424030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:45979 (local address 127.0.0.1:46478)
0923 17:27:15.424189 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:15.424241 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.60:45979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260923 17:27:15.424430 25593 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.229.60:45979: Network error: Client connection negotiation failed: client connection to 127.11.229.60:45979: connect: Connection refused (error 111)
Sep 23 17:27:15 dist-test-slave-ptl9 krb5kdc[25512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184435, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.61@KRBTEST.COM for kudu/127.11.229.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.60.keytab.
Entry for principal kudu/127.11.229.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.60.keytab.
I20260923 17:27:15.431300 25672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.423212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48578 (local address 127.11.229.61:45601)
0923 17:27:15.423335 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:15.423339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:15.423353 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:27:15.423389 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:15.423392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:15.423433 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:15.423514 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:15.424018 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.424788 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.425668 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.425953 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.426037 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:15.428240 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:15.428242 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:15.428245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:15.428266 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:15.429646 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:15.430062 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:15.430064 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:15.430066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:15.430144 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:15.430425 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:15.430427 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:15.430429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:15.430605 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:15.430678 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:15.430940 (+ 262us) server_negotiation.cc:299] Negotiation successful
0923 17:27:15.431098 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":51}
I20260923 17:27:15.431360 25671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.423002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:45601 (local address 127.0.0.1:48578)
0923 17:27:15.423158 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:15.423175 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:15.423256 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:15.423562 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:15.423568 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:15.423770 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:15.423915 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:15.423925 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.424889 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.424892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:15.425559 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:15.425569 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.426057 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.426059 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:15.426137 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:15.426686 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:15.426698 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:15.428114 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:15.429742 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:15.429745 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:15.429747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:15.429977 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:15.430228 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:15.430230 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:15.430231 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:15.430343 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:15.430695 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:15.430697 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:15.430773 (+ 76us) client_negotiation.cc:769] Sending connection context
0923 17:27:15.430855 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 17:27:15.431045 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260923 17:27:15.432055 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.60:45979
--webserver_interface=127.11.229.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.60.keytab
--principal=kudu/127.11.229.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:43247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:43351,127.11.229.61:45601,127.11.229.60:45979
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:27:15.433331 25668 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45979 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:27:15.434082 25671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.433834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:45979 (local address 127.0.0.1:46484)
0923 17:27:15.433951 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:15.434026 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.60:45979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260923 17:27:15.434648 25616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.229.60:45979: connect: Connection refused (error 111)
I20260923 17:27:15.435532 25671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.435326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:45979 (local address 127.0.0.1:46496)
0923 17:27:15.435433 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:15.435483 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.60:45979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260923 17:27:15.435698 25668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.229.60:45979: Network error: Client connection negotiation failed: client connection to 127.11.229.60:45979: connect: Connection refused (error 111)
I20260923 17:27:15.455605 25593 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45979 } attempt: 1
I20260923 17:27:15.456538 25596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.456228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:45979 (local address 127.0.0.1:46506)
0923 17:27:15.456394 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:15.456459 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.60:45979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260923 17:27:15.457365 25596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.457078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:45979 (local address 127.0.0.1:46520)
0923 17:27:15.457193 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:15.457253 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.60:45979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260923 17:27:15.457494 25593 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.229.60:45979: Network error: Client connection negotiation failed: client connection to 127.11.229.60:45979: connect: Connection refused (error 111)
I20260923 17:27:15.484969 25668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45979 } attempt: 1
I20260923 17:27:15.485989 25671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.485666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:45979 (local address 127.0.0.1:46526)
0923 17:27:15.485838 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:15.485903 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.60:45979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260923 17:27:15.486789 25671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.486544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:45979 (local address 127.0.0.1:46534)
0923 17:27:15.486688 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:15.486732 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.60:45979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260923 17:27:15.486922 25668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.229.60:45979: Network error: Client connection negotiation failed: client connection to 127.11.229.60:45979: connect: Connection refused (error 111)
I20260923 17:27:15.532651 25593 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45979 } attempt: 2
I20260923 17:27:15.533551 25596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.533227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:45979 (local address 127.0.0.1:46544)
0923 17:27:15.533411 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:15.533473 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.60:45979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260923 17:27:15.534294 25596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.534081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:45979 (local address 127.0.0.1:46560)
0923 17:27:15.534198 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:15.534246 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.60:45979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260923 17:27:15.534417 25593 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.229.60:45979: Network error: Client connection negotiation failed: client connection to 127.11.229.60:45979: connect: Connection refused (error 111)
W20260923 17:27:15.546062 25674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:15.546351 25674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:15.546453 25674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:15.550642 25674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:27:15.550721 25674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:15.550760 25674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:15.550819 25674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:27:15.550855 25674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:27:15.555119 25668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45979 } attempt: 2
I20260923 17:27:15.555974 25671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.555673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:45979 (local address 127.0.0.1:46570)
0923 17:27:15.555823 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:15.555892 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.60:45979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260923 17:27:15.556373 25674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:43247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:43351,127.11.229.61:45601,127.11.229.60:45979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.60.keytab
--principal=kudu/127.11.229.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.60:45979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.229.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:27:15.556736 25671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.556496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:45979 (local address 127.0.0.1:46580)
0923 17:27:15.556617 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:15.556669 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.60:45979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260923 17:27:15.556883 25668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.229.60:45979: Network error: Client connection negotiation failed: client connection to 127.11.229.60:45979: connect: Connection refused (error 111)
I20260923 17:27:15.557614 25674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:15.558684 25674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:15.567166 25683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:15.567163 25681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:15.567203 25680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:15.567441 25674 server_base.cc:1034] running on GCE node
I20260923 17:27:15.568107 25674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:15.569212 25674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:15.570436 25674 hybrid_clock.cc:648] HybridClock initialized: now 1790184435570401 us; error 47 us; skew 500 ppm
Sep 23 17:27:15 dist-test-slave-ptl9 krb5kdc[25512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184435, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:15.573115 25674 init.cc:377] Logged in from keytab as kudu/127.11.229.60@KRBTEST.COM (short username kudu)
I20260923 17:27:15.574076 25674 webserver.cc:460] Webserver started at http://127.11.229.60:46719/ using document root <none> and password file <none>
I20260923 17:27:15.574649 25674 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:15.574738 25674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:15.574960 25674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:27:15.576974 25674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-2/data/instance:
uuid: "da1ed5391bd543fd9817292b64e99561"
format_stamp: "Formatted at 2026-09-23 17:27:15 on dist-test-slave-ptl9"
server_key: "5e4528de0489a3346758f77a90048a08"
server_key_iv: "801a22df1d4ac00547245c004b6dc03c"
server_key_version: "encryptionkey@0"
I20260923 17:27:15.577575 25674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-2/wal/instance:
uuid: "da1ed5391bd543fd9817292b64e99561"
format_stamp: "Formatted at 2026-09-23 17:27:15 on dist-test-slave-ptl9"
server_key: "5e4528de0489a3346758f77a90048a08"
server_key_iv: "801a22df1d4ac00547245c004b6dc03c"
server_key_version: "encryptionkey@0"
I20260923 17:27:15.581146 25674 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:27:15.583532 25690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:15.584672 25674 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:27:15.584816 25674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-2/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-2/wal
uuid: "da1ed5391bd543fd9817292b64e99561"
format_stamp: "Formatted at 2026-09-23 17:27:15 on dist-test-slave-ptl9"
server_key: "5e4528de0489a3346758f77a90048a08"
server_key_iv: "801a22df1d4ac00547245c004b6dc03c"
server_key_version: "encryptionkey@0"
I20260923 17:27:15.584926 25674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:15.610347 25674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:15.613253 25674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:15.613530 25674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:15.620882 25674 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.60:45979
I20260923 17:27:15.620906 25742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.60:45979 every 8 connection(s)
I20260923 17:27:15.621905 25674 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-2/data/info.pb
I20260923 17:27:15.624651 25743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:27:15.628939 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 25674
I20260923 17:27:15.629063 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184271597484-12180-0/minicluster-data/master-2/wal/instance
I20260923 17:27:15.628716 25743 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.229.62" port: 43351 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:27:15.629391 12180 external_mini_cluster.cc:1428] Setting key 746f02f42ea3891e4d72dd50ba2ea022
I20260923 17:27:15.633576 25593 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45979 } attempt: 3
Sep 23 17:27:15 dist-test-slave-ptl9 krb5kdc[25512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184435, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for kudu/127.11.229.60@KRBTEST.COM
Sep 23 17:27:15 dist-test-slave-ptl9 krb5kdc[25512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.60@KRBTEST.COM
Sep 23 17:27:15 dist-test-slave-ptl9 krb5kdc[25512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184435, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.60@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:27:15.643873 25596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.633967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:45979 (local address 127.0.0.1:46594)
0923 17:27:15.634080 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:15.634094 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:15.634171 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:15.634679 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:15.634682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:15.634862 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:15.634951 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:15.634959 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.635854 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.635861 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:15.636633 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:15.636644 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.637126 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.637130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:15.637212 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:15.637728 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:15.637741 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:15.638956 (+ 1215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:15.641685 (+ 2729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:15.641688 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:15.641690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:15.642075 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:15.642449 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:15.642452 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:15.642454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:15.642550 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:15.643346 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:15.643350 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:15.643445 (+ 95us) client_negotiation.cc:769] Sending connection context
0923 17:27:15.643553 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 17:27:15.643692 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260923 17:27:15.644392 25595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.630718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51944 (local address 127.11.229.62:43351)
0923 17:27:15.630878 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:15.630887 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:15.631498 (+ 611us) server_negotiation.cc:407] Connection header received
0923 17:27:15.632352 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:15.632355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:15.632403 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:15.632482 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:15.633505 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.634237 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.635608 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.635938 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.636040 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:15.640330 (+ 4290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:15.640333 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:15.640336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:15.640360 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:15.641990 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:15.642505 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:15.642509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:15.642510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:15.642589 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:15.643129 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:15.643133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:15.643135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:15.643451 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:15.643559 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:15.644183 (+ 624us) server_negotiation.cc:299] Negotiation successful
0923 17:27:15.644289 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57,"spinlock_wait_cycles":768}
I20260923 17:27:15.645440 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.630781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:43351 (local address 127.0.0.1:51944)
0923 17:27:15.631341 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:15.631392 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:15.632123 (+ 731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:15.632562 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:15.632567 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:15.633015 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:15.633360 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:15.633381 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.634341 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.634344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:15.635470 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:15.635480 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.636089 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.636093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:15.636257 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:15.637718 (+ 1461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:15.637751 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:15.640187 (+ 2436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:15.642097 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:15.642106 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:15.642117 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:15.642422 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:15.642694 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:15.642697 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:15.642699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:15.642806 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:15.643559 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:15.643567 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:15.643707 (+ 140us) client_negotiation.cc:769] Sending connection context
0923 17:27:15.644554 (+ 847us) client_negotiation.cc:240] Negotiation successful
0923 17:27:15.644747 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":43,"thread_start_us":142,"threads_started":1}
I20260923 17:27:15.645448 25747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.630781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46584 (local address 127.11.229.60:45979)
0923 17:27:15.631286 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:15.631305 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:15.631345 (+ 40us) server_negotiation.cc:407] Connection header received
0923 17:27:15.632123 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:15.632148 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:15.632384 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:15.632573 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:15.633408 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.634634 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.635640 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.636188 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.636300 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:15.639734 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:15.639739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:15.639742 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:15.639777 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:15.642205 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:15.642701 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:15.642705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:15.642706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:15.642811 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:15.643099 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:15.643101 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:15.643102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:15.643250 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:15.643977 (+ 727us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:15.644105 (+ 128us) server_negotiation.cc:299] Negotiation successful
0923 17:27:15.644491 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":119,"server-negotiator.queue_time_us":396,"spinlock_wait_cycles":51968,"thread_start_us":145,"threads_started":1}
I20260923 17:27:15.645448 25748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.634175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46594 (local address 127.11.229.60:45979)
0923 17:27:15.634476 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:15.634483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:15.634494 (+ 11us) server_negotiation.cc:407] Connection header received
0923 17:27:15.634522 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:15.634524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:15.634554 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:15.634654 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:15.635072 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.635751 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.636767 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.637012 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.637120 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:15.639105 (+ 1985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:15.639113 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:15.639130 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:15.639162 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:15.641556 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:15.642186 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:15.642195 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:15.642198 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:15.642340 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:15.642658 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:15.642662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:15.642666 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:15.643214 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:15.643360 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:15.644173 (+ 813us) server_negotiation.cc:299] Negotiation successful
0923 17:27:15.644442 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":247,"thread_start_us":100,"threads_started":1}
I20260923 17:27:15.647375 25743 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.229.61" port: 45601 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:27:15.648090 25668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45979 } attempt: 3
I20260923 17:27:15.650051 25593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa4b4079ec574eaca7d0b8b679017021: Bootstrap starting.
I20260923 17:27:15.652890 25593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa4b4079ec574eaca7d0b8b679017021: Neither blocks nor log segments found. Creating new log.
Sep 23 17:27:15 dist-test-slave-ptl9 krb5kdc[25512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184435, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.61@KRBTEST.COM for kudu/127.11.229.60@KRBTEST.COM
I20260923 17:27:15.653903 25593 log.cc:824] T 00000000000000000000000000000000 P fa4b4079ec574eaca7d0b8b679017021: Log is configured to *not* fsync() on all Append() calls
Sep 23 17:27:15 dist-test-slave-ptl9 krb5kdc[25512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
Sep 23 17:27:15 dist-test-slave-ptl9 krb5kdc[25512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184435, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.60@KRBTEST.COM for kudu/127.11.229.61@KRBTEST.COM
I20260923 17:27:15.657756 25593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa4b4079ec574eaca7d0b8b679017021: No bootstrap required, opened a new log
I20260923 17:27:15.658200 25671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.648426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:45979 (local address 127.0.0.1:46598)
0923 17:27:15.648577 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:15.648592 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:15.648670 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:15.648887 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:15.648893 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:15.649077 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:15.649202 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:15.649209 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.650282 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.650286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:15.651007 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:15.651018 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.651649 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.651653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:15.652062 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:15.652434 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:15.652452 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:15.653987 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:15.656362 (+ 2375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:15.656366 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:15.656371 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:15.656678 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:15.657005 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:15.657008 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:15.657010 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:15.657106 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:15.657590 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:15.657594 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:15.657701 (+ 107us) client_negotiation.cc:769] Sending connection context
0923 17:27:15.657872 (+ 171us) client_negotiation.cc:240] Negotiation successful
0923 17:27:15.658022 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":321}
I20260923 17:27:15.658731 25595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.647898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51950 (local address 127.11.229.62:43351)
0923 17:27:15.648001 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:15.648005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:15.648334 (+ 329us) server_negotiation.cc:407] Connection header received
0923 17:27:15.648377 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:15.648382 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:15.648428 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:15.648499 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:15.649354 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.650101 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.651170 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.651501 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.651584 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:15.654736 (+ 3152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:15.654749 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:15.654752 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:15.654777 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:15.656451 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:15.656949 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:15.656953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:15.656956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:15.657036 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:15.657404 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:15.657407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:15.657409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:15.657623 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:15.658381 (+ 758us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:15.658464 (+ 83us) server_negotiation.cc:299] Negotiation successful
0923 17:27:15.658584 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 17:27:15.659075 25748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.648482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46598 (local address 127.11.229.60:45979)
0923 17:27:15.648696 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:15.648700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:15.648710 (+ 10us) server_negotiation.cc:407] Connection header received
0923 17:27:15.648739 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:15.648742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:15.648781 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:15.648866 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:15.649370 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.650070 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.651164 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.651517 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.651721 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:15.654142 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:15.654145 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:15.654148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:15.654177 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:15.656224 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:15.656806 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:15.656809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:15.656811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:15.656896 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:15.657222 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:15.657226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:15.657227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:15.657477 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:15.658486 (+ 1009us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:15.658560 (+ 74us) server_negotiation.cc:299] Negotiation successful
0923 17:27:15.658665 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260923 17:27:15.659781 25672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.648179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48586 (local address 127.11.229.61:45601)
0923 17:27:15.648303 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:15.648309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:15.648355 (+ 46us) server_negotiation.cc:407] Connection header received
0923 17:27:15.649006 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:15.649012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:15.649064 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:15.649116 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:15.649754 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.650468 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.651489 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.651830 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.651917 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:15.655587 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:15.655593 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:15.655596 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:15.655623 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:15.657219 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:15.657896 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:15.657899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:15.657901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:15.657984 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:15.658543 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:15.658547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:15.658548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:15.658751 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:15.658823 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:15.659508 (+ 685us) server_negotiation.cc:299] Negotiation successful
0923 17:27:15.659621 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":57,"spinlock_wait_cycles":16256}
I20260923 17:27:15.660398 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.647867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:45601 (local address 127.0.0.1:48586)
0923 17:27:15.648292 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:15.648307 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:15.648556 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:15.649355 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:15.649358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:15.649516 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:15.649623 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:15.649629 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.650600 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.650603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:15.651367 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:15.651376 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.652299 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.652302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:15.653395 (+ 1093us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:15.653925 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:15.653940 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:15.655423 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:15.657376 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:15.657382 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:15.657386 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:15.657731 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:15.658103 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:15.658109 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:15.658112 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:15.658215 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:15.659000 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:15.659003 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:15.659089 (+ 86us) client_negotiation.cc:769] Sending connection context
0923 17:27:15.660097 (+ 1008us) client_negotiation.cc:240] Negotiation successful
0923 17:27:15.660251 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345}
I20260923 17:27:15.661237 25743 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45979 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:27:15.662142 25593 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa4b4079ec574eaca7d0b8b679017021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4b4079ec574eaca7d0b8b679017021" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 43351 } } peers { permanent_uuid: "a8a9ae00dc0149fda4b630578d134ae1" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 45601 } } peers { permanent_uuid: "da1ed5391bd543fd9817292b64e99561" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45979 } }
I20260923 17:27:15.662420 25593 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa4b4079ec574eaca7d0b8b679017021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:27:15.662477 25593 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa4b4079ec574eaca7d0b8b679017021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa4b4079ec574eaca7d0b8b679017021, State: Initialized, Role: FOLLOWER
I20260923 17:27:15.662510 25668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8a9ae00dc0149fda4b630578d134ae1: Bootstrap starting.
I20260923 17:27:15.662972 25593 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa4b4079ec574eaca7d0b8b679017021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4b4079ec574eaca7d0b8b679017021" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 43351 } } peers { permanent_uuid: "a8a9ae00dc0149fda4b630578d134ae1" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 45601 } } peers { permanent_uuid: "da1ed5391bd543fd9817292b64e99561" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45979 } }
I20260923 17:27:15.663525 25750 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa4b4079ec574eaca7d0b8b679017021 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4b4079ec574eaca7d0b8b679017021" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 43351 } } peers { permanent_uuid: "a8a9ae00dc0149fda4b630578d134ae1" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 45601 } } peers { permanent_uuid: "da1ed5391bd543fd9817292b64e99561" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45979 } } }
I20260923 17:27:15.663726 25750 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa4b4079ec574eaca7d0b8b679017021 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:27:15.664490 25593 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa4b4079ec574eaca7d0b8b679017021 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:27:15.665387 25668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8a9ae00dc0149fda4b630578d134ae1: Neither blocks nor log segments found. Creating new log.
I20260923 17:27:15.666254 25668 log.cc:824] T 00000000000000000000000000000000 P a8a9ae00dc0149fda4b630578d134ae1: Log is configured to *not* fsync() on all Append() calls
Sep 23 17:27:15 dist-test-slave-ptl9 krb5kdc[25512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.61@KRBTEST.COM
I20260923 17:27:15.669078 25668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8a9ae00dc0149fda4b630578d134ae1: No bootstrap required, opened a new log
W20260923 17:27:15.670912 25762 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fa4b4079ec574eaca7d0b8b679017021: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:27:15.670989 25762 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:27:15.671243 25672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.660280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48602 (local address 127.11.229.61:45601)
0923 17:27:15.660621 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:15.660624 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:15.660636 (+ 12us) server_negotiation.cc:407] Connection header received
0923 17:27:15.660666 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:15.660669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:15.660708 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:15.660786 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:15.661682 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.662396 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.663465 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.663800 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.663914 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:15.667167 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:15.667174 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:15.667178 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:15.667216 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:15.669296 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:15.669802 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:15.669806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:15.669808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:15.669888 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:15.670191 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:15.670194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:15.670196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:15.670419 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:15.670567 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:15.670948 (+ 381us) server_negotiation.cc:299] Negotiation successful
0923 17:27:15.671071 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277}
I20260923 17:27:15.673311 25668 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8a9ae00dc0149fda4b630578d134ae1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4b4079ec574eaca7d0b8b679017021" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 43351 } } peers { permanent_uuid: "a8a9ae00dc0149fda4b630578d134ae1" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 45601 } } peers { permanent_uuid: "da1ed5391bd543fd9817292b64e99561" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45979 } }
I20260923 17:27:15.673537 25668 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8a9ae00dc0149fda4b630578d134ae1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:27:15.673590 25668 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8a9ae00dc0149fda4b630578d134ae1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8a9ae00dc0149fda4b630578d134ae1, State: Initialized, Role: FOLLOWER
I20260923 17:27:15.674196 25668 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8a9ae00dc0149fda4b630578d134ae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4b4079ec574eaca7d0b8b679017021" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 43351 } } peers { permanent_uuid: "a8a9ae00dc0149fda4b630578d134ae1" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 45601 } } peers { permanent_uuid: "da1ed5391bd543fd9817292b64e99561" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45979 } }
I20260923 17:27:15.675184 12180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:27:15.671274 25525 master_runner.cc:186] Error getting master registration for 127.11.229.61:45601: OK, instance_id { permanent_uuid: "a8a9ae00dc0149fda4b630578d134ae1" instance_seqno: 1790184435391246 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 17:27:15.675398 25525 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 17:27:15.675623 25668 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8a9ae00dc0149fda4b630578d134ae1 [sys.catalog]: configured and running, proceeding with master startup.
Sep 23 17:27:15 dist-test-slave-ptl9 krb5kdc[25512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184435, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.60@KRBTEST.COM for kudu/127.11.229.60@KRBTEST.COM
W20260923 17:27:15.681541 25779 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a8a9ae00dc0149fda4b630578d134ae1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:27:15.681666 25779 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:27:15.682567 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.666958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:45979 (local address 127.0.0.1:46600)
0923 17:27:15.667131 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:15.667145 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:15.667221 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:15.667708 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:15.667710 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:15.667882 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:15.667975 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:15.667984 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.675490 (+ 7506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.675493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:15.676302 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:15.676315 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.676985 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.676988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:15.677057 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:15.677610 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:15.677629 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:15.679185 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:15.681067 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:15.681070 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:15.681073 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:15.681324 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:15.681588 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:15.681590 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:15.681592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:15.681687 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:15.682087 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:15.682090 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:15.682182 (+ 92us) client_negotiation.cc:769] Sending connection context
0923 17:27:15.682272 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 17:27:15.682407 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260923 17:27:15.683238 25748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.667397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46600 (local address 127.11.229.60:45979)
0923 17:27:15.667506 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:15.667510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:15.667523 (+ 13us) server_negotiation.cc:407] Connection header received
0923 17:27:15.667564 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:15.667570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:15.667613 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:15.667698 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:15.669333 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.670081 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.676446 (+ 6365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.676767 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.676870 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:15.679317 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:15.679320 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:15.679322 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:15.679343 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:15.680953 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:15.681423 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:15.681426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:15.681427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:15.681499 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:15.681783 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:15.681787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:15.681789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:15.681985 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:15.682098 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:15.682965 (+ 867us) server_negotiation.cc:299] Negotiation successful
0923 17:27:15.683077 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260923 17:27:15.684757 25764 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8a9ae00dc0149fda4b630578d134ae1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4b4079ec574eaca7d0b8b679017021" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 43351 } } peers { permanent_uuid: "a8a9ae00dc0149fda4b630578d134ae1" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 45601 } } peers { permanent_uuid: "da1ed5391bd543fd9817292b64e99561" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45979 } } }
I20260923 17:27:15.684911 25764 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8a9ae00dc0149fda4b630578d134ae1 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:27:15.686568 25743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da1ed5391bd543fd9817292b64e99561: Bootstrap starting.
I20260923 17:27:15.684299 25600 master_runner.cc:186] Error getting master registration for 127.11.229.60:45979: OK, instance_id { permanent_uuid: "da1ed5391bd543fd9817292b64e99561" instance_seqno: 1790184435618853 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 17:27:15.687975 25600 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 17:27:15.689520 25743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da1ed5391bd543fd9817292b64e99561: Neither blocks nor log segments found. Creating new log.
I20260923 17:27:15.690476 25743 log.cc:824] T 00000000000000000000000000000000 P da1ed5391bd543fd9817292b64e99561: Log is configured to *not* fsync() on all Append() calls
I20260923 17:27:15.701675 25672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.683211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48618 (local address 127.11.229.61:45601)
0923 17:27:15.683322 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:15.683326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:15.690382 (+ 7056us) server_negotiation.cc:407] Connection header received
0923 17:27:15.692040 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:15.692044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:15.692101 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:15.692207 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:15.693238 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.694117 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.695336 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.695696 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.695772 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:15.697741 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:15.697745 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:15.697748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:15.697777 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:15.699550 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:15.700259 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:15.700263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:15.700265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:15.700348 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:15.700643 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:15.700646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:15.700649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:15.700860 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:15.700977 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:15.701375 (+ 398us) server_negotiation.cc:299] Negotiation successful
0923 17:27:15.701502 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 17:27:15.707537 25743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da1ed5391bd543fd9817292b64e99561: No bootstrap required, opened a new log
I20260923 17:27:15.708256 25747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.682352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46604 (local address 127.11.229.60:45979)
0923 17:27:15.682460 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:15.682464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:15.690147 (+ 7683us) server_negotiation.cc:407] Connection header received
0923 17:27:15.690763 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:15.690767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:15.690817 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:15.690871 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:15.695153 (+ 4282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.695957 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.698082 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.698487 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.698568 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:15.700904 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:15.700908 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:15.700911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:15.700939 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:15.703013 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:15.706198 (+ 3185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:15.706202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:15.706204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:15.706305 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:15.707275 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:15.707279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:15.707282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:15.707502 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:15.707607 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:15.707931 (+ 324us) server_negotiation.cc:299] Negotiation successful
0923 17:27:15.708058 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 17:27:15.708835 25595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:15.681679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51954 (local address 127.11.229.62:43351)
0923 17:27:15.681790 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:15.681795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:15.689942 (+ 8147us) server_negotiation.cc:407] Connection header received
0923 17:27:15.692048 (+ 2106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:15.692052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:15.692107 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:15.692159 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:15.693101 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.693946 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.697091 (+ 3145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:15.697475 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:15.697556 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:15.701319 (+ 3763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:15.701325 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:15.701328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:15.701362 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:15.703761 (+ 2399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:15.705797 (+ 2036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:15.705801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:15.705803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:15.705892 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:15.706597 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:15.706601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:15.706603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:15.706829 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:15.706942 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:15.708552 (+ 1610us) server_negotiation.cc:299] Negotiation successful
0923 17:27:15.708680 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 17:27:15.712560 25743 raft_consensus.cc:348] T 00000000000000000000000000000000 P da1ed5391bd543fd9817292b64e99561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4b4079ec574eaca7d0b8b679017021" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 43351 } } peers { permanent_uuid: "a8a9ae00dc0149fda4b630578d134ae1" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 45601 } } peers { permanent_uuid: "da1ed5391bd543fd9817292b64e99561" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45979 } }
I20260923 17:27:15.712833 25743 raft_consensus.cc:374] T 00000000000000000000000000000000 P da1ed5391bd543fd9817292b64e99561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:27:15.712899 25743 raft_consensus.cc:729] T 00000000000000000000000000000000 P da1ed5391bd543fd9817292b64e99561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da1ed5391bd543fd9817292b64e99561, State: Initialized, Role: FOLLOWER
I20260923 17:27:15.713477 25743 consensus_queue.cc:260] T 00000000000000000000000000000000 P da1ed5391bd543fd9817292b64e99561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4b4079ec574eaca7d0b8b679017021" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 43351 } } peers { permanent_uuid: "a8a9ae00dc0149fda4b630578d134ae1" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 45601 } } peers { permanent_uuid: "da1ed5391bd543fd9817292b64e99561" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45979 } }
I20260923 17:27:15.715049 25743 sys_catalog.cc:564] T 00000000000000000000000000000000 P da1ed5391bd543fd9817292b64e99561 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:27:15.715073 25785 sys_catalog.cc:455] T 00000000000000000000000000000000 P da1ed5391bd543fd9817292b64e99561 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4b4079ec574eaca7d0b8b679017021" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 43351 } } peers { permanent_uuid: "a8a9ae00dc0149fda4b630578d134ae1" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 45601 } } peers { permanent_uuid: "da1ed5391bd543fd9817292b64e99561" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45979 } } }
I20260923 17:27:15.715332 25785 sys_catalog.cc:458] T 00000000000000000000000000000000 P da1ed5391bd543fd9817292b64e99561 [sys.catalog]: This master's current role is: FOLLOWER
W20260923 17:27:15.721016 25796 catalog_manager.cc:1558] T 00000000000000000000000000000000 P da1ed5391bd543fd9817292b64e99561: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:27:15.721107 25796 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:27:15.726308 25674 master_runner.cc:263] No leader master found from master da1ed5391bd543fd9817292b64e99561
I20260923 17:27:15.726377 25674 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 17:27:15.756678 25764 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8a9ae00dc0149fda4b630578d134ae1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:27:15.756881 25764 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8a9ae00dc0149fda4b630578d134ae1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4b4079ec574eaca7d0b8b679017021" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 43351 } } peers { permanent_uuid: "a8a9ae00dc0149fda4b630578d134ae1" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 45601 } } peers { permanent_uuid: "da1ed5391bd543fd9817292b64e99561" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45979 } }
I20260923 17:27:15.757992 25764 leader_election.cc:288] T 00000000000000000000000000000000 P a8a9ae00dc0149fda4b630578d134ae1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa4b4079ec574eaca7d0b8b679017021 (127.11.229.62:43351), da1ed5391bd543fd9817292b64e99561 (127.11.229.60:45979)
I20260923 17:27:15.758831 25567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a8a9ae00dc0149fda4b630578d134ae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa4b4079ec574eaca7d0b8b679017021" is_pre_election: true
I20260923 17:27:15.759150 25567 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fa4b4079ec574eaca7d0b8b679017021 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8a9ae00dc0149fda4b630578d134ae1 in term 0.
I20260923 17:27:15.759940 25717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a8a9ae00dc0149fda4b630578d134ae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da1ed5391bd543fd9817292b64e99561" is_pre_election: true
I20260923 17:27:15.760248 25717 raft_consensus.cc:2454] T 00000000000000000000000000000000 P da1ed5391bd543fd9817292b64e99561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8a9ae00dc0149fda4b630578d134ae1 in term 0.
I20260923 17:27:15.760725 25616 leader_election.cc:302] T 00000000000000000000000000000000 P a8a9ae00dc0149fda4b630578d134ae1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8a9ae00dc0149fda4b630578d134ae1, da1ed5391bd543fd9817292b64e99561; no voters:
I20260923 17:27:15.761291 25764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8a9ae00dc0149fda4b630578d134ae1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:27:15.761375 25764 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8a9ae00dc0149fda4b630578d134ae1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:27:15.761417 25764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8a9ae00dc0149fda4b630578d134ae1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:27:15.762781 25764 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8a9ae00dc0149fda4b630578d134ae1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4b4079ec574eaca7d0b8b679017021" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 43351 } } peers { permanent_uuid: "a8a9ae00dc0149fda4b630578d134ae1" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 45601 } } peers { permanent_uuid: "da1ed5391bd543fd9817292b64e99561" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45979 } }
I20260923 17:27:15.763088 25764 leader_election.cc:288] T 00000000000000000000000000000000 P a8a9ae00dc0149fda4b630578d134ae1 [CANDIDATE]: Term 1 election: Requested vote from peers fa4b4079ec574eaca7d0b8b679017021 (127.11.229.62:43351), da1ed5391bd543fd9817292b64e99561 (127.11.229.60:45979)
I20260923 17:27:15.763676 25567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a8a9ae00dc0149fda4b630578d134ae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa4b4079ec574eaca7d0b8b679017021"
I20260923 17:27:15.763803 25567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa4b4079ec574eaca7d0b8b679017021 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:27:15.765555 25567 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fa4b4079ec574eaca7d0b8b679017021 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8a9ae00dc0149fda4b630578d134ae1 in term 1.
I20260923 17:27:15.766036 25717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a8a9ae00dc0149fda4b630578d134ae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da1ed5391bd543fd9817292b64e99561"
I20260923 17:27:15.766149 25717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da1ed5391bd543fd9817292b64e99561 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:27:15.767392 25717 raft_consensus.cc:2454] T 00000000000000000000000000000000 P da1ed5391bd543fd9817292b64e99561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8a9ae00dc0149fda4b630578d134ae1 in term 1.
I20260923 17:27:15.767759 25616 leader_election.cc:302] T 00000000000000000000000000000000 P a8a9ae00dc0149fda4b630578d134ae1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8a9ae00dc0149fda4b630578d134ae1, da1ed5391bd543fd9817292b64e99561; no voters:
I20260923 17:27:15.770251 25764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8a9ae00dc0149fda4b630578d134ae1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:27:15.770565 25764 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8a9ae00dc0149fda4b630578d134ae1 [term 1 LEADER]: Becoming Leader. State: Replica: a8a9ae00dc0149fda4b630578d134ae1, State: Running, Role: LEADER
I20260923 17:27:15.770977 25764 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8a9ae00dc0149fda4b630578d134ae1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4b4079ec574eaca7d0b8b679017021" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 43351 } } peers { permanent_uuid: "a8a9ae00dc0149fda4b630578d134ae1" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 45601 } } peers { permanent_uuid: "da1ed5391bd543fd9817292b64e99561" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45979 } }
I20260923 17:27:15.772890 25764 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8a9ae00dc0149fda4b630578d134ae1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8a9ae00dc0149fda4b630578d134ae1. Latest consensus state: current_term: 1 leader_uuid: "a8a9ae00dc0149fda4b630578d134ae1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4b4079ec574eaca7d0b8b679017021" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 43351 } } peers { permanent_uuid: "a8a9ae00dc0149fda4b630578d134ae1" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 45601 } } peers { permanent_uuid: "da1ed5391bd543fd9817292b64e99561" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45979 } } }
I20260923 17:27:15.773077 25764 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8a9ae00dc0149fda4b630578d134ae1 [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:15.773543 25800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:27:15.774534 25800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:27:15.785416 25717 raft_consensus.cc:1261] T 00000000000000000000000000000000 P da1ed5391bd543fd9817292b64e99561 [term 1 FOLLOWER]: Refusing update from remote peer a8a9ae00dc0149fda4b630578d134ae1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:27:15.786167 25764 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a8a9ae00dc0149fda4b630578d134ae1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da1ed5391bd543fd9817292b64e99561" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:27:15.791162 25785 sys_catalog.cc:455] T 00000000000000000000000000000000 P da1ed5391bd543fd9817292b64e99561 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8a9ae00dc0149fda4b630578d134ae1. Latest consensus state: current_term: 1 leader_uuid: "a8a9ae00dc0149fda4b630578d134ae1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4b4079ec574eaca7d0b8b679017021" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 43351 } } peers { permanent_uuid: "a8a9ae00dc0149fda4b630578d134ae1" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 45601 } } peers { permanent_uuid: "da1ed5391bd543fd9817292b64e99561" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45979 } } }
I20260923 17:27:15.791308 25785 sys_catalog.cc:458] T 00000000000000000000000000000000 P da1ed5391bd543fd9817292b64e99561 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:27:15.795099 25567 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fa4b4079ec574eaca7d0b8b679017021 [term 1 FOLLOWER]: Refusing update from remote peer a8a9ae00dc0149fda4b630578d134ae1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:27:15.795893 25764 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a8a9ae00dc0149fda4b630578d134ae1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa4b4079ec574eaca7d0b8b679017021" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 43351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:27:15.800146 25750 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa4b4079ec574eaca7d0b8b679017021 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8a9ae00dc0149fda4b630578d134ae1. Latest consensus state: current_term: 1 leader_uuid: "a8a9ae00dc0149fda4b630578d134ae1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4b4079ec574eaca7d0b8b679017021" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 43351 } } peers { permanent_uuid: "a8a9ae00dc0149fda4b630578d134ae1" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 45601 } } peers { permanent_uuid: "da1ed5391bd543fd9817292b64e99561" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45979 } } }
I20260923 17:27:15.800285 25750 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa4b4079ec574eaca7d0b8b679017021 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:27:15.808317 25798 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8a9ae00dc0149fda4b630578d134ae1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a8a9ae00dc0149fda4b630578d134ae1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4b4079ec574eaca7d0b8b679017021" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 43351 } } peers { permanent_uuid: "a8a9ae00dc0149fda4b630578d134ae1" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 45601 } } peers { permanent_uuid: "da1ed5391bd543fd9817292b64e99561" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45979 } } }
I20260923 17:27:15.808454 25798 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8a9ae00dc0149fda4b630578d134ae1 [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:15.809644 25750 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa4b4079ec574eaca7d0b8b679017021 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a8a9ae00dc0149fda4b630578d134ae1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4b4079ec574eaca7d0b8b679017021" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 43351 } } peers { permanent_uuid: "a8a9ae00dc0149fda4b630578d134ae1" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 45601 } } peers { permanent_uuid: "da1ed5391bd543fd9817292b64e99561" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45979 } } }
I20260923 17:27:15.809762 25750 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa4b4079ec574eaca7d0b8b679017021 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:27:15.810209 25801 mvcc.cc:205] Tried to move back new op lower bound from 7332595448969347072 to 7332595448921481216. Current Snapshot: MvccSnapshot[applied={T|T < 7332595448969347072}]
I20260923 17:27:15.811378 25804 mvcc.cc:205] Tried to move back new op lower bound from 7332595448969347072 to 7332595448921481216. Current Snapshot: MvccSnapshot[applied={T|T < 7332595448969347072}]
I20260923 17:27:15.814252 25764 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8a9ae00dc0149fda4b630578d134ae1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a8a9ae00dc0149fda4b630578d134ae1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4b4079ec574eaca7d0b8b679017021" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 43351 } } peers { permanent_uuid: "a8a9ae00dc0149fda4b630578d134ae1" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 45601 } } peers { permanent_uuid: "da1ed5391bd543fd9817292b64e99561" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45979 } } }
I20260923 17:27:15.814370 25764 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8a9ae00dc0149fda4b630578d134ae1 [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:15.815595 25785 sys_catalog.cc:455] T 00000000000000000000000000000000 P da1ed5391bd543fd9817292b64e99561 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a8a9ae00dc0149fda4b630578d134ae1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4b4079ec574eaca7d0b8b679017021" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 43351 } } peers { permanent_uuid: "a8a9ae00dc0149fda4b630578d134ae1" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 45601 } } peers { permanent_uuid: "da1ed5391bd543fd9817292b64e99561" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 45979 } } }
I20260923 17:27:15.815709 25785 sys_catalog.cc:458] T 00000000000000000000000000000000 P da1ed5391bd543fd9817292b64e99561 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:27:15.816936 25802 mvcc.cc:205] Tried to move back new op lower bound from 7332595448969347072 to 7332595448921481216. Current Snapshot: MvccSnapshot[applied={T|T < 7332595448969347072}]
I20260923 17:27:15.819324 25800 catalog_manager.cc:1347] Generated new cluster ID: a83d7122e98d43caae9729b6440c029c
I20260923 17:27:15.819387 25800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:27:15.866552 25800 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:27:15.867915 25800 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:27:15.880200 25800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8a9ae00dc0149fda4b630578d134ae1: Generated new TSK 0
I20260923 17:27:15.880935 25800 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:27:16.672457 25762 catalog_manager.cc:1259] Loaded cluster ID: a83d7122e98d43caae9729b6440c029c
I20260923 17:27:16.672597 25762 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fa4b4079ec574eaca7d0b8b679017021: loading cluster ID for follower catalog manager: success
I20260923 17:27:16.674739 25762 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fa4b4079ec574eaca7d0b8b679017021: acquiring CA information for follower catalog manager: success
I20260923 17:27:16.676365 25762 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fa4b4079ec574eaca7d0b8b679017021: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 17:27:16.722559 25796 catalog_manager.cc:1259] Loaded cluster ID: a83d7122e98d43caae9729b6440c029c
I20260923 17:27:16.722657 25796 catalog_manager.cc:1552] T 00000000000000000000000000000000 P da1ed5391bd543fd9817292b64e99561: loading cluster ID for follower catalog manager: success
I20260923 17:27:16.724304 25796 catalog_manager.cc:1574] T 00000000000000000000000000000000 P da1ed5391bd543fd9817292b64e99561: acquiring CA information for follower catalog manager: success
I20260923 17:27:16.725487 25796 catalog_manager.cc:1602] T 00000000000000000000000000000000 P da1ed5391bd543fd9817292b64e99561: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 17:27:16.951917 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 25525
I20260923 17:27:16.958840 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 25600
I20260923 17:27:16.964908 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 25674
2026-09-23T17:27:16Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3177 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:27:17 dist-test-slave-ptl9 krb5kdc[25814](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:27:17 dist-test-slave-ptl9 krb5kdc[25814](info): set up 2 sockets
Sep 23 17:27:17 dist-test-slave-ptl9 krb5kdc[25814](info): commencing operation
krb5kdc: starting...
W20260923 17:27:18.005666 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:27:18 dist-test-slave-ptl9 krb5kdc[25814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:27:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:27:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.229.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
I20260923 17:27:18.144381 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:38665
--webserver_interface=127.11.229.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:43565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:38665
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:27:18.255498 25827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:18.255755 25827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:18.255795 25827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:18.259763 25827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:27:18.259817 25827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:18.259836 25827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:18.259852 25827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:27:18.259867 25827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:27:18.264993 25827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:43565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:38665
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:38665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:27:18.266031 25827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:18.267020 25827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:18.275483 25835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:18.275624 25827 server_base.cc:1034] running on GCE node
W20260923 17:27:18.275511 25833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:18.275483 25832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:18.276357 25827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:18.277544 25827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:18.278729 25827 hybrid_clock.cc:648] HybridClock initialized: now 1790184438278703 us; error 43 us; skew 500 ppm
Sep 23 17:27:18 dist-test-slave-ptl9 krb5kdc[25814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184438, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:18.281440 25827 init.cc:377] Logged in from keytab as kudu/127.11.229.62@KRBTEST.COM (short username kudu)
I20260923 17:27:18.282276 25827 webserver.cc:460] Webserver started at http://127.11.229.62:42395/ using document root <none> and password file <none>
I20260923 17:27:18.282840 25827 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:18.282927 25827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:18.283138 25827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:27:18.285140 25827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "a94e60d183014be195398f39d61041d9"
format_stamp: "Formatted at 2026-09-23 17:27:18 on dist-test-slave-ptl9"
server_key: "d3100bd183b4abbc516db87e8805f0ad"
server_key_iv: "3d75e57bae425f8c2aefe14caae60a8f"
server_key_version: "encryptionkey@0"
I20260923 17:27:18.285776 25827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "a94e60d183014be195398f39d61041d9"
format_stamp: "Formatted at 2026-09-23 17:27:18 on dist-test-slave-ptl9"
server_key: "d3100bd183b4abbc516db87e8805f0ad"
server_key_iv: "3d75e57bae425f8c2aefe14caae60a8f"
server_key_version: "encryptionkey@0"
I20260923 17:27:18.289499 25827 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:27:18.291973 25842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:18.293171 25827 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:27:18.293340 25827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "a94e60d183014be195398f39d61041d9"
format_stamp: "Formatted at 2026-09-23 17:27:18 on dist-test-slave-ptl9"
server_key: "d3100bd183b4abbc516db87e8805f0ad"
server_key_iv: "3d75e57bae425f8c2aefe14caae60a8f"
server_key_version: "encryptionkey@0"
I20260923 17:27:18.293459 25827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:18.303478 25827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:18.306579 25827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:18.306793 25827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:18.313824 25827 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:38665
I20260923 17:27:18.313838 25894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:38665 every 8 connection(s)
I20260923 17:27:18.314767 25827 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:27:18.317721 25895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:27:18.320636 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 25827
I20260923 17:27:18.320772 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:27:18.321086 12180 external_mini_cluster.cc:1428] Setting key f93a21fba99e81967b479254a22fda87
I20260923 17:27:18.324195 25895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a94e60d183014be195398f39d61041d9: Bootstrap starting.
I20260923 17:27:18.326999 25895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a94e60d183014be195398f39d61041d9: Neither blocks nor log segments found. Creating new log.
I20260923 17:27:18.327847 25895 log.cc:824] T 00000000000000000000000000000000 P a94e60d183014be195398f39d61041d9: Log is configured to *not* fsync() on all Append() calls
I20260923 17:27:18.329918 25895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a94e60d183014be195398f39d61041d9: No bootstrap required, opened a new log
Sep 23 17:27:18 dist-test-slave-ptl9 krb5kdc[25814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:27:18.333384 25895 raft_consensus.cc:348] T 00000000000000000000000000000000 P a94e60d183014be195398f39d61041d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a94e60d183014be195398f39d61041d9" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 38665 } }
I20260923 17:27:18.333633 25895 raft_consensus.cc:374] T 00000000000000000000000000000000 P a94e60d183014be195398f39d61041d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:27:18.333724 25895 raft_consensus.cc:729] T 00000000000000000000000000000000 P a94e60d183014be195398f39d61041d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a94e60d183014be195398f39d61041d9, State: Initialized, Role: FOLLOWER
I20260923 17:27:18.334259 25895 consensus_queue.cc:260] T 00000000000000000000000000000000 P a94e60d183014be195398f39d61041d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a94e60d183014be195398f39d61041d9" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 38665 } }
I20260923 17:27:18.334434 25895 raft_consensus.cc:388] T 00000000000000000000000000000000 P a94e60d183014be195398f39d61041d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:27:18.334506 25895 raft_consensus.cc:482] T 00000000000000000000000000000000 P a94e60d183014be195398f39d61041d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:27:18.334622 25895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a94e60d183014be195398f39d61041d9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:27:18.335767 25895 raft_consensus.cc:504] T 00000000000000000000000000000000 P a94e60d183014be195398f39d61041d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a94e60d183014be195398f39d61041d9" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 38665 } }
I20260923 17:27:18.336126 25895 leader_election.cc:302] T 00000000000000000000000000000000 P a94e60d183014be195398f39d61041d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a94e60d183014be195398f39d61041d9; no voters:
I20260923 17:27:18.336462 25895 leader_election.cc:288] T 00000000000000000000000000000000 P a94e60d183014be195398f39d61041d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:27:18.336575 25900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a94e60d183014be195398f39d61041d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:27:18.336778 25900 raft_consensus.cc:686] T 00000000000000000000000000000000 P a94e60d183014be195398f39d61041d9 [term 1 LEADER]: Becoming Leader. State: Replica: a94e60d183014be195398f39d61041d9, State: Running, Role: LEADER
I20260923 17:27:18.336906 25898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:18.322443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53124 (local address 127.11.229.62:38665)
0923 17:27:18.322864 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:18.322885 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:18.322934 (+ 49us) server_negotiation.cc:407] Connection header received
0923 17:27:18.323762 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:18.323786 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:18.324201 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:18.324565 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:18.325237 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:18.326778 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:18.327682 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:18.328388 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:18.328571 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:18.331246 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:18.331252 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:18.331266 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:18.331293 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:18.333620 (+ 2327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:18.334175 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:18.334180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:18.334184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:18.334295 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:18.334600 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:18.334603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:18.334605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:18.335087 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:18.335247 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:18.335538 (+ 291us) server_negotiation.cc:299] Negotiation successful
0923 17:27:18.335848 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":99,"threads_started":1}
I20260923 17:27:18.337198 25900 consensus_queue.cc:237] T 00000000000000000000000000000000 P a94e60d183014be195398f39d61041d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a94e60d183014be195398f39d61041d9" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 38665 } }
I20260923 17:27:18.337991 25895 sys_catalog.cc:564] T 00000000000000000000000000000000 P a94e60d183014be195398f39d61041d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:27:18.338542 25900 sys_catalog.cc:455] T 00000000000000000000000000000000 P a94e60d183014be195398f39d61041d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a94e60d183014be195398f39d61041d9. Latest consensus state: current_term: 1 leader_uuid: "a94e60d183014be195398f39d61041d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a94e60d183014be195398f39d61041d9" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 38665 } } }
I20260923 17:27:18.338655 25900 sys_catalog.cc:458] T 00000000000000000000000000000000 P a94e60d183014be195398f39d61041d9 [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:18.339027 25904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:27:18.338999 25901 sys_catalog.cc:455] T 00000000000000000000000000000000 P a94e60d183014be195398f39d61041d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a94e60d183014be195398f39d61041d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a94e60d183014be195398f39d61041d9" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 38665 } } }
I20260923 17:27:18.339104 25901 sys_catalog.cc:458] T 00000000000000000000000000000000 P a94e60d183014be195398f39d61041d9 [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:18.340265 12180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:27:18.343132 25904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:27:18.349500 25904 catalog_manager.cc:1347] Generated new cluster ID: 65ddb5b802864e2aabf6bd56f351043b
I20260923 17:27:18.349586 25904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:27:18.360637 25904 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:27:18.362051 25904 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:27:18.374114 25904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a94e60d183014be195398f39d61041d9: Generated new TSK 0
I20260923 17:27:18.375052 25904 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:27:18.396443 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 25827
2026-09-23T17:27:18Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1430 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:27:18 dist-test-slave-ptl9 krb5kdc[25922](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:27:18 dist-test-slave-ptl9 krb5kdc[25922](info): set up 2 sockets
Sep 23 17:27:18 dist-test-slave-ptl9 krb5kdc[25922](info): commencing operation
krb5kdc: starting...
W20260923 17:27:19.433794 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:27:19 dist-test-slave-ptl9 krb5kdc[25922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:27:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:27:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.229.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
I20260923 17:27:19.555784 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:41427
--webserver_interface=127.11.229.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:44585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:41427
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:27:19.668766 25935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:19.669036 25935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:19.669078 25935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:19.673301 25935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:27:19.673367 25935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:19.673385 25935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:19.673403 25935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:27:19.673463 25935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:27:19.678851 25935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:44585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:41427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:41427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:27:19.679944 25935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:19.681001 25935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 17:27:19.689599 25935 server_base.cc:1034] running on GCE node
W20260923 17:27:19.689544 25941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:19.689568 25940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:19.689544 25943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:19.690333 25935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:19.691462 25935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:19.692657 25935 hybrid_clock.cc:648] HybridClock initialized: now 1790184439692645 us; error 28 us; skew 500 ppm
Sep 23 17:27:19 dist-test-slave-ptl9 krb5kdc[25922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184439, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:19.701495 25935 init.cc:377] Logged in from keytab as kudu/127.11.229.62@KRBTEST.COM (short username kudu)
I20260923 17:27:19.702564 25935 webserver.cc:460] Webserver started at http://127.11.229.62:43819/ using document root <none> and password file <none>
I20260923 17:27:19.703333 25935 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:19.703459 25935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:19.703696 25935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:27:19.706019 25935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "c863213e6d3941c2890d8e505df89311"
format_stamp: "Formatted at 2026-09-23 17:27:19 on dist-test-slave-ptl9"
server_key: "f145fb9a8be9e7b499670d19cf3e76c7"
server_key_iv: "2e902bb5d573604c3c2ee4cff81d027e"
server_key_version: "encryptionkey@0"
I20260923 17:27:19.706632 25935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "c863213e6d3941c2890d8e505df89311"
format_stamp: "Formatted at 2026-09-23 17:27:19 on dist-test-slave-ptl9"
server_key: "f145fb9a8be9e7b499670d19cf3e76c7"
server_key_iv: "2e902bb5d573604c3c2ee4cff81d027e"
server_key_version: "encryptionkey@0"
I20260923 17:27:19.710563 25935 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:27:19.713101 25950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:19.714389 25935 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:27:19.714536 25935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "c863213e6d3941c2890d8e505df89311"
format_stamp: "Formatted at 2026-09-23 17:27:19 on dist-test-slave-ptl9"
server_key: "f145fb9a8be9e7b499670d19cf3e76c7"
server_key_iv: "2e902bb5d573604c3c2ee4cff81d027e"
server_key_version: "encryptionkey@0"
I20260923 17:27:19.714654 25935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:19.733799 25935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:19.737046 25935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:19.737323 25935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:19.745016 25935 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:41427
I20260923 17:27:19.745378 26002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:41427 every 8 connection(s)
I20260923 17:27:19.746141 25935 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:27:19.749073 26003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:27:19.752439 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 25935
I20260923 17:27:19.752547 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:27:19.753012 12180 external_mini_cluster.cc:1428] Setting key db6fd1b0a1c3cd9eb34d2733e5145ced
I20260923 17:27:19.755416 26003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311: Bootstrap starting.
I20260923 17:27:19.758249 26003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311: Neither blocks nor log segments found. Creating new log.
I20260923 17:27:19.759137 26003 log.cc:824] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311: Log is configured to *not* fsync() on all Append() calls
I20260923 17:27:19.761492 26003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311: No bootstrap required, opened a new log
Sep 23 17:27:19 dist-test-slave-ptl9 krb5kdc[25922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:27:19.765290 26003 raft_consensus.cc:348] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c863213e6d3941c2890d8e505df89311" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 41427 } }
I20260923 17:27:19.765580 26003 raft_consensus.cc:374] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:27:19.765655 26003 raft_consensus.cc:729] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c863213e6d3941c2890d8e505df89311, State: Initialized, Role: FOLLOWER
I20260923 17:27:19.766150 26003 consensus_queue.cc:260] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c863213e6d3941c2890d8e505df89311" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 41427 } }
I20260923 17:27:19.766347 26003 raft_consensus.cc:388] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:27:19.766417 26003 raft_consensus.cc:482] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:27:19.766516 26003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:27:19.767382 26003 raft_consensus.cc:504] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c863213e6d3941c2890d8e505df89311" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 41427 } }
I20260923 17:27:19.767726 26003 leader_election.cc:302] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c863213e6d3941c2890d8e505df89311; no voters:
I20260923 17:27:19.768075 26003 leader_election.cc:288] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:27:19.768190 26008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:27:19.768414 26008 raft_consensus.cc:686] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [term 1 LEADER]: Becoming Leader. State: Replica: c863213e6d3941c2890d8e505df89311, State: Running, Role: LEADER
I20260923 17:27:19.768757 26008 consensus_queue.cc:237] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c863213e6d3941c2890d8e505df89311" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 41427 } }
I20260923 17:27:19.769418 26003 sys_catalog.cc:564] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:27:19.769690 26006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:19.754562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41438 (local address 127.11.229.62:41427)
0923 17:27:19.754995 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:19.755022 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:19.755065 (+ 43us) server_negotiation.cc:407] Connection header received
0923 17:27:19.755905 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:19.755930 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:19.756357 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:19.756724 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:19.757505 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:19.759033 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:19.760164 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:19.760824 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:19.760990 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:19.763568 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:19.763574 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:19.763592 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:19.763625 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:19.766120 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:19.766681 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:19.766687 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:19.766694 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:19.766826 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:19.767113 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:19.767117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:19.767119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:19.767551 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:19.767766 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:19.768409 (+ 643us) server_negotiation.cc:299] Negotiation successful
0923 17:27:19.768718 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":121,"threads_started":1}
I20260923 17:27:19.771011 26009 sys_catalog.cc:455] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c863213e6d3941c2890d8e505df89311" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c863213e6d3941c2890d8e505df89311" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 41427 } } }
I20260923 17:27:19.771158 26009 sys_catalog.cc:458] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:19.771412 26010 sys_catalog.cc:455] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c863213e6d3941c2890d8e505df89311. Latest consensus state: current_term: 1 leader_uuid: "c863213e6d3941c2890d8e505df89311" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c863213e6d3941c2890d8e505df89311" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 41427 } } }
I20260923 17:27:19.771493 26010 sys_catalog.cc:458] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:19.772761 26017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:27:19.775300 26017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:27:19.780352 26017 catalog_manager.cc:1347] Generated new cluster ID: 66a085f4ceb9403bbb8e345b2a1ad23f
I20260923 17:27:19.780413 26017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:27:19.785148 26017 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:27:19.786028 26017 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:27:19.790313 26017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311: Generated new TSK 0
I20260923 17:27:19.790935 26017 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.229.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab.
Entry for principal kudu/127.11.229.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab.
I20260923 17:27:19.825424 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.1:0
--local_ip_for_outbound_sockets=127.11.229.1
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:41427
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:44585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:27:19.940171 26029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:19.940447 26029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:19.940492 26029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:19.944846 26029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:19.944906 26029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:19.945003 26029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.1
I20260923 17:27:19.950598 26029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:44585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:41427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.1
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:27:19.951799 26029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:19.952996 26029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 17:27:19.963356 26029 server_base.cc:1034] running on GCE node
W20260923 17:27:19.963294 26034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:19.963306 26035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:19.963325 26037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:19.963984 26029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:19.964668 26029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:19.965871 26029 hybrid_clock.cc:648] HybridClock initialized: now 1790184439965844 us; error 37 us; skew 500 ppm
Sep 23 17:27:19 dist-test-slave-ptl9 krb5kdc[25922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184439, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:19.968590 26029 init.cc:377] Logged in from keytab as kudu/127.11.229.1@KRBTEST.COM (short username kudu)
I20260923 17:27:19.969534 26029 webserver.cc:460] Webserver started at http://127.11.229.1:34821/ using document root <none> and password file <none>
I20260923 17:27:19.970075 26029 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:19.970132 26029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:19.970295 26029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:27:19.972517 26029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/data/instance:
uuid: "300a1144ecb7434a9e1dd68677cab6cf"
format_stamp: "Formatted at 2026-09-23 17:27:19 on dist-test-slave-ptl9"
server_key: "3cae6515633ab0f53fe1de8dace8a170"
server_key_iv: "975ef45b8786364365e90e24a6b7f52c"
server_key_version: "encryptionkey@0"
I20260923 17:27:19.973090 26029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance:
uuid: "300a1144ecb7434a9e1dd68677cab6cf"
format_stamp: "Formatted at 2026-09-23 17:27:19 on dist-test-slave-ptl9"
server_key: "3cae6515633ab0f53fe1de8dace8a170"
server_key_iv: "975ef45b8786364365e90e24a6b7f52c"
server_key_version: "encryptionkey@0"
I20260923 17:27:19.976783 26029 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:27:19.979195 26044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:19.980454 26029 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:27:19.980580 26029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/wal
uuid: "300a1144ecb7434a9e1dd68677cab6cf"
format_stamp: "Formatted at 2026-09-23 17:27:19 on dist-test-slave-ptl9"
server_key: "3cae6515633ab0f53fe1de8dace8a170"
server_key_iv: "975ef45b8786364365e90e24a6b7f52c"
server_key_version: "encryptionkey@0"
I20260923 17:27:19.980685 26029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:19.999787 26029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:20.003176 26029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:20.003394 26029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:20.004150 26029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:27:20.005128 26029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:27:20.005177 26029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:20.005219 26029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:27:20.005300 26029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260923 17:27:20.014798 26029 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.1:43095
I20260923 17:27:20.014818 26157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.1:43095 every 8 connection(s)
I20260923 17:27:20.015880 26029 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
I20260923 17:27:20.022154 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 26029
I20260923 17:27:20.022312 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance
I20260923 17:27:20.022609 12180 external_mini_cluster.cc:1428] Setting key 16844f3f49109adf15cbf4a786c28b5a
Sep 23 17:27:20 dist-test-slave-ptl9 krb5kdc[25922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184439, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:27:20.031293 26006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:20.017878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:40555 (local address 127.11.229.62:41427)
0923 17:27:20.018029 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:20.018035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:20.019154 (+ 1119us) server_negotiation.cc:407] Connection header received
0923 17:27:20.020570 (+ 1416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:20.020572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:20.020621 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:20.020701 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:20.022019 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:20.022660 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:20.024297 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:20.024539 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:20.024655 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:20.027531 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:20.027534 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:20.027536 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:20.027560 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:20.029093 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:20.029676 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:20.029680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:20.029682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:20.029774 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:20.030134 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:20.030137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:20.030139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:20.030337 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:20.030445 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:20.030960 (+ 515us) server_negotiation.cc:299] Negotiation successful
0923 17:27:20.031092 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.11.229.2@KRBTEST.COM; defaulting to no policy
I20260923 17:27:20.032151 26160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:20.018383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.1:40555)
0923 17:27:20.018955 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:20.019025 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:20.020358 (+ 1333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:20.020883 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:20.020889 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:20.021252 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:20.021843 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:20.021857 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:20.022812 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:20.022816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:20.024157 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:20.024169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:20.024706 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:20.024710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:20.024898 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:20.025617 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:20.025652 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:20.027346 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:20.029236 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:20.029242 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:20.029292 (+ 50us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:20.029565 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:20.029871 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:20.029880 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:20.029882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:20.030029 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:20.030444 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:20.030452 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:20.030705 (+ 253us) client_negotiation.cc:769] Sending connection context
0923 17:27:20.030904 (+ 199us) client_negotiation.cc:240] Negotiation successful
0923 17:27:20.031239 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":131,"threads_started":1}
I20260923 17:27:20.033514 26158 heartbeater.cc:347] Connected to a master server at 127.11.229.62:41427
I20260923 17:27:20.033788 26158 heartbeater.cc:464] Registering TS with master...
I20260923 17:27:20.034399 26158 heartbeater.cc:511] Master 127.11.229.62:41427 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.2@KRBTEST.COM" created.
I20260923 17:27:20.036219 25967 ts_manager.cc:194] Registered new tserver with Master: 300a1144ecb7434a9e1dd68677cab6cf (127.11.229.1:43095)
I20260923 17:27:20.037511 25967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.1@KRBTEST.COM'} at 127.11.229.1:40555
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab.
Entry for principal kudu/127.11.229.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab.
I20260923 17:27:20.052773 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.2:0
--local_ip_for_outbound_sockets=127.11.229.2
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:41427
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:44585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:27:20.164165 26163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:20.164449 26163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:20.164525 26163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:20.168506 26163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:20.168592 26163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:20.168679 26163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.2
I20260923 17:27:20.174489 26163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:44585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:41427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.2
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:27:20.175594 26163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:20.176642 26163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:20.186681 26169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:20.186673 26168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:20.186673 26171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:20.186939 26163 server_base.cc:1034] running on GCE node
I20260923 17:27:20.187409 26163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:20.188041 26163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:20.189242 26163 hybrid_clock.cc:648] HybridClock initialized: now 1790184440189222 us; error 30 us; skew 500 ppm
Sep 23 17:27:20 dist-test-slave-ptl9 krb5kdc[25922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184440, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:20.192070 26163 init.cc:377] Logged in from keytab as kudu/127.11.229.2@KRBTEST.COM (short username kudu)
I20260923 17:27:20.192941 26163 webserver.cc:460] Webserver started at http://127.11.229.2:35055/ using document root <none> and password file <none>
I20260923 17:27:20.193591 26163 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:20.193684 26163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:20.193921 26163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:27:20.195978 26163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/data/instance:
uuid: "73d1d9b41a914836a80114437655ec6e"
format_stamp: "Formatted at 2026-09-23 17:27:20 on dist-test-slave-ptl9"
server_key: "4431f6672aa0ddca05bc42c366860378"
server_key_iv: "beae22808722dc86821f544c122d1c4b"
server_key_version: "encryptionkey@0"
I20260923 17:27:20.196540 26163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance:
uuid: "73d1d9b41a914836a80114437655ec6e"
format_stamp: "Formatted at 2026-09-23 17:27:20 on dist-test-slave-ptl9"
server_key: "4431f6672aa0ddca05bc42c366860378"
server_key_iv: "beae22808722dc86821f544c122d1c4b"
server_key_version: "encryptionkey@0"
I20260923 17:27:20.200424 26163 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 17:27:20.203068 26178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:20.204329 26163 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:27:20.204474 26163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/wal
uuid: "73d1d9b41a914836a80114437655ec6e"
format_stamp: "Formatted at 2026-09-23 17:27:20 on dist-test-slave-ptl9"
server_key: "4431f6672aa0ddca05bc42c366860378"
server_key_iv: "beae22808722dc86821f544c122d1c4b"
server_key_version: "encryptionkey@0"
I20260923 17:27:20.204592 26163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:20.236011 26163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:20.239621 26163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:20.239840 26163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:20.240466 26163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:27:20.241355 26163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:27:20.241428 26163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:20.241503 26163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:27:20.241557 26163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:20.250692 26163 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.2:42075
I20260923 17:27:20.250717 26291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.2:42075 every 8 connection(s)
I20260923 17:27:20.251633 26163 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
I20260923 17:27:20.259449 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 26163
I20260923 17:27:20.259564 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance
I20260923 17:27:20.259853 12180 external_mini_cluster.cc:1428] Setting key 6e1bdc4d008af7e02f9668e94cac2952
Sep 23 17:27:20 dist-test-slave-ptl9 krb5kdc[25922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184440, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:27:20.266721 26006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:20.253644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:50343 (local address 127.11.229.62:41427)
0923 17:27:20.253795 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:20.253799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:20.254660 (+ 861us) server_negotiation.cc:407] Connection header received
0923 17:27:20.256085 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:20.256089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:20.256151 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:20.256263 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:20.257524 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:20.258083 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:20.259301 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:20.259586 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:20.259738 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:20.262743 (+ 3005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:20.262746 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:20.262748 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:20.262772 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:20.264273 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:20.264852 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:20.264859 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:20.264863 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:20.264965 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:20.265365 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:20.265371 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:20.265375 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:20.265664 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:20.265778 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:20.266317 (+ 539us) server_negotiation.cc:299] Negotiation successful
0923 17:27:20.266471 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 17:27:20.267500 26294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:20.254056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.2:50343)
0923 17:27:20.254503 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:20.254547 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:20.255852 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:20.256458 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:20.256465 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:20.256836 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:20.257371 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:20.257385 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:20.258221 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:20.258230 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:20.259129 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:20.259139 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:20.259788 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:20.259792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:20.260022 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:20.260662 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:20.260678 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:20.262561 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:20.264398 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:20.264406 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:20.264432 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:20.264737 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:20.265062 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:20.265064 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:20.265066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:20.265199 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:20.265773 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:20.265778 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:20.266034 (+ 256us) client_negotiation.cc:769] Sending connection context
0923 17:27:20.266199 (+ 165us) client_negotiation.cc:240] Negotiation successful
0923 17:27:20.266501 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":123,"threads_started":1}
I20260923 17:27:20.268934 26292 heartbeater.cc:347] Connected to a master server at 127.11.229.62:41427
I20260923 17:27:20.269184 26292 heartbeater.cc:464] Registering TS with master...
I20260923 17:27:20.269755 26292 heartbeater.cc:511] Master 127.11.229.62:41427 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.229.3@KRBTEST.COM; defaulting to no policy
I20260923 17:27:20.270985 25967 ts_manager.cc:194] Registered new tserver with Master: 73d1d9b41a914836a80114437655ec6e (127.11.229.2:42075)
I20260923 17:27:20.271624 25967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.2@KRBTEST.COM'} at 127.11.229.2:50343
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab.
Entry for principal kudu/127.11.229.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab.
I20260923 17:27:20.289057 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.3:0
--local_ip_for_outbound_sockets=127.11.229.3
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:41427
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:44585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:27:20.401542 26297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:20.401809 26297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:20.401850 26297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:20.406013 26297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:20.406101 26297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:20.406214 26297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.3
I20260923 17:27:20.411626 26297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:44585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:41427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.3
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:27:20.412746 26297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:20.414076 26297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:20.424041 26302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:20.424171 26297 server_base.cc:1034] running on GCE node
W20260923 17:27:20.423995 26303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:20.423995 26305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:20.424746 26297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:20.425469 26297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:20.426658 26297 hybrid_clock.cc:648] HybridClock initialized: now 1790184440426641 us; error 27 us; skew 500 ppm
Sep 23 17:27:20 dist-test-slave-ptl9 krb5kdc[25922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184440, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:20.429356 26297 init.cc:377] Logged in from keytab as kudu/127.11.229.3@KRBTEST.COM (short username kudu)
I20260923 17:27:20.430244 26297 webserver.cc:460] Webserver started at http://127.11.229.3:42641/ using document root <none> and password file <none>
I20260923 17:27:20.430764 26297 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:20.430822 26297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:20.430981 26297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:27:20.432946 26297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/data/instance:
uuid: "de8c684e115544cfa24e6fd295f49c9c"
format_stamp: "Formatted at 2026-09-23 17:27:20 on dist-test-slave-ptl9"
server_key: "3baced430a71467226be102e91ac1056"
server_key_iv: "99e00b927367ff9332dedf682ce6f80e"
server_key_version: "encryptionkey@0"
I20260923 17:27:20.433542 26297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance:
uuid: "de8c684e115544cfa24e6fd295f49c9c"
format_stamp: "Formatted at 2026-09-23 17:27:20 on dist-test-slave-ptl9"
server_key: "3baced430a71467226be102e91ac1056"
server_key_iv: "99e00b927367ff9332dedf682ce6f80e"
server_key_version: "encryptionkey@0"
I20260923 17:27:20.437050 26297 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:27:20.439421 26312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:20.440623 26297 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:27:20.440750 26297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/wal
uuid: "de8c684e115544cfa24e6fd295f49c9c"
format_stamp: "Formatted at 2026-09-23 17:27:20 on dist-test-slave-ptl9"
server_key: "3baced430a71467226be102e91ac1056"
server_key_iv: "99e00b927367ff9332dedf682ce6f80e"
server_key_version: "encryptionkey@0"
I20260923 17:27:20.440857 26297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:20.454010 26297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:20.457003 26297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:20.457211 26297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:20.457887 26297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:27:20.458739 26297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:27:20.458787 26297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:20.458828 26297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:27:20.458885 26297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:20.468659 26297 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.3:35255
I20260923 17:27:20.471951 26425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.3:35255 every 8 connection(s)
I20260923 17:27:20.473242 26297 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
I20260923 17:27:20.479743 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 26297
I20260923 17:27:20.479884 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance
I20260923 17:27:20.480257 12180 external_mini_cluster.cc:1428] Setting key 1186c769205b6c580c943a04bb863a7c
Sep 23 17:27:20 dist-test-slave-ptl9 krb5kdc[25922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184440, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:27:20.492851 26006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:20.476646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:52169 (local address 127.11.229.62:41427)
0923 17:27:20.476839 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:20.476844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:20.477989 (+ 1145us) server_negotiation.cc:407] Connection header received
0923 17:27:20.479389 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:20.479392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:20.479478 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:20.479554 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:20.481814 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:20.482596 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:20.484511 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:20.484928 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:20.485059 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:20.488628 (+ 3569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:20.488632 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:20.488635 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:20.488671 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:20.490543 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:20.491220 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:20.491226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:20.491230 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:20.491326 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:20.491706 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:20.491709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:20.491711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:20.491911 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:20.492022 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:20.492548 (+ 526us) server_negotiation.cc:299] Negotiation successful
0923 17:27:20.492678 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 17:27:20.494335 26428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:20.477233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.3:52169)
0923 17:27:20.477836 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:20.477881 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:20.479153 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:20.479769 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:20.479782 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:20.480381 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:20.481166 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:20.481193 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:20.482832 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:20.482838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:20.484346 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:20.484379 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:20.485157 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:20.485162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:20.485436 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:20.486161 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:20.486184 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:20.488435 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:20.490680 (+ 2245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:20.490686 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:20.490703 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:20.491104 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:20.491407 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:20.491410 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:20.491412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:20.491588 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:20.492022 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:20.492030 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:20.492299 (+ 269us) client_negotiation.cc:769] Sending connection context
0923 17:27:20.492494 (+ 195us) client_negotiation.cc:240] Negotiation successful
0923 17:27:20.493166 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":158,"threads_started":1}
I20260923 17:27:20.495651 26426 heartbeater.cc:347] Connected to a master server at 127.11.229.62:41427
I20260923 17:27:20.495910 26426 heartbeater.cc:464] Registering TS with master...
I20260923 17:27:20.496387 26426 heartbeater.cc:511] Master 127.11.229.62:41427 requested a full tablet report, sending...
I20260923 17:27:20.497466 25967 ts_manager.cc:194] Registered new tserver with Master: de8c684e115544cfa24e6fd295f49c9c (127.11.229.3:35255)
I20260923 17:27:20.498046 25967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.3@KRBTEST.COM'} at 127.11.229.3:52169
I20260923 17:27:20.506172 12180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:27:20.515189 26006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:20.508115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41454 (local address 127.11.229.62:41427)
0923 17:27:20.508247 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:20.508250 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:20.508386 (+ 136us) server_negotiation.cc:407] Connection header received
0923 17:27:20.508491 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:20.508493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:20.508530 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:20.508595 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:20.509378 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:20.509882 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:20.510700 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:20.510911 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:20.510991 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:20.512270 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:20.512273 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:20.512275 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:20.512298 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:20.513778 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:20.514170 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:20.514173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:20.514174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:20.514247 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:20.514473 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:20.514475 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:20.514476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:20.514680 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:20.514780 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:20.514949 (+ 169us) server_negotiation.cc:299] Negotiation successful
0923 17:27:20.515036 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 17:27:20.518107 25967 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41454:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:27:20.521078 25967 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:27:20.535087 26438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:20.529479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:42075 (local address 127.0.0.1:46950)
0923 17:27:20.530298 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:20.530347 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:20.530462 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:20.531270 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:20.531275 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:20.531296 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:27:20.531433 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:20.531442 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:20.532529 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:20.532533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:20.533745 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:20.533754 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:20.534546 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:20.534550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:20.534654 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:20.534690 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 17:27:20.534832 (+ 142us) client_negotiation.cc:240] Negotiation successful
0923 17:27:20.534949 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":72,"threads_started":1}
I20260923 17:27:20.535105 26439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:20.529595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46950 (local address 127.11.229.2:42075)
0923 17:27:20.530771 (+ 1176us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:20.530783 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:20.530808 (+ 25us) server_negotiation.cc:407] Connection header received
0923 17:27:20.530891 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:20.530896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:20.531110 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:20.531276 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:27:20.531560 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:20.532394 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:20.533867 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:20.534425 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:20.534548 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:20.534701 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:20.534861 (+ 160us) server_negotiation.cc:299] Negotiation successful
0923 17:27:20.534953 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1053,"thread_start_us":152,"threads_started":1}
I20260923 17:27:20.535988 26440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:20.529523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:35255 (local address 127.0.0.1:52264)
0923 17:27:20.530674 (+ 1151us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:20.530688 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:20.530777 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:20.531450 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:20.531453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:20.531470 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:27:20.531608 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:20.531614 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:20.533051 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:20.533054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:20.534433 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:20.534445 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:20.535601 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:20.535605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:20.535697 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:20.535729 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 17:27:20.535802 (+ 73us) client_negotiation.cc:240] Negotiation successful
0923 17:27:20.535865 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1089,"mutex_wait_us":63,"thread_start_us":117,"threads_started":1}
I20260923 17:27:20.536137 26441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:20.530173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52264 (local address 127.11.229.3:35255)
0923 17:27:20.530789 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:20.530802 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:20.530828 (+ 26us) server_negotiation.cc:407] Connection header received
0923 17:27:20.530904 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:20.530911 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:20.531124 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:20.531271 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:27:20.531734 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:20.532900 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:20.534578 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:20.535387 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:20.535507 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:20.535656 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:20.535858 (+ 202us) server_negotiation.cc:299] Negotiation successful
0923 17:27:20.535966 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":119,"threads_started":1}
I20260923 17:27:20.537346 26437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:20.529795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:43095 (local address 127.0.0.1:39960)
0923 17:27:20.530916 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:20.530929 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:20.531015 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:20.531456 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:20.531458 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:20.531480 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:27:20.531629 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:20.531636 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:20.533185 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:20.533188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:20.534433 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:20.534445 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:20.535976 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:20.535979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:20.536987 (+ 1008us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:20.537020 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 17:27:20.537114 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 17:27:20.537186 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1066,"mutex_wait_us":84,"spinlock_wait_cycles":1280,"thread_start_us":75,"threads_started":1}
I20260923 17:27:20.537403 26442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:20.530294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39960 (local address 127.11.229.1:43095)
0923 17:27:20.530596 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:20.530611 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:20.531023 (+ 412us) server_negotiation.cc:407] Connection header received
0923 17:27:20.531112 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:20.531116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:20.531276 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:20.531404 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:27:20.531802 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:20.533054 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:20.534590 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:20.535641 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:20.535721 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:20.535877 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:20.537148 (+ 1271us) server_negotiation.cc:299] Negotiation successful
0923 17:27:20.537237 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":80,"threads_started":1}
I20260923 17:27:20.538618 26341 tablet_service.cc:1467] Processing CreateTablet for tablet 02a47f27fd4f4079a5b14381752d0f85 (DEFAULT_TABLE table=test-table [id=e57eb4a8d79c412b83814002504f0a39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:27:20.538618 26226 tablet_service.cc:1467] Processing CreateTablet for tablet 02a47f27fd4f4079a5b14381752d0f85 (DEFAULT_TABLE table=test-table [id=e57eb4a8d79c412b83814002504f0a39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:27:20.539762 26341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02a47f27fd4f4079a5b14381752d0f85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:27:20.539758 26226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02a47f27fd4f4079a5b14381752d0f85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:27:20.539471 26092 tablet_service.cc:1467] Processing CreateTablet for tablet 02a47f27fd4f4079a5b14381752d0f85 (DEFAULT_TABLE table=test-table [id=e57eb4a8d79c412b83814002504f0a39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:27:20.540325 26092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02a47f27fd4f4079a5b14381752d0f85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:27:20.546826 26443 tablet_bootstrap.cc:492] T 02a47f27fd4f4079a5b14381752d0f85 P 300a1144ecb7434a9e1dd68677cab6cf: Bootstrap starting.
I20260923 17:27:20.548017 26445 tablet_bootstrap.cc:492] T 02a47f27fd4f4079a5b14381752d0f85 P 73d1d9b41a914836a80114437655ec6e: Bootstrap starting.
I20260923 17:27:20.549415 26443 tablet_bootstrap.cc:654] T 02a47f27fd4f4079a5b14381752d0f85 P 300a1144ecb7434a9e1dd68677cab6cf: Neither blocks nor log segments found. Creating new log.
I20260923 17:27:20.549616 26444 tablet_bootstrap.cc:492] T 02a47f27fd4f4079a5b14381752d0f85 P de8c684e115544cfa24e6fd295f49c9c: Bootstrap starting.
I20260923 17:27:20.550357 26443 log.cc:824] T 02a47f27fd4f4079a5b14381752d0f85 P 300a1144ecb7434a9e1dd68677cab6cf: Log is configured to *not* fsync() on all Append() calls
I20260923 17:27:20.550693 26445 tablet_bootstrap.cc:654] T 02a47f27fd4f4079a5b14381752d0f85 P 73d1d9b41a914836a80114437655ec6e: Neither blocks nor log segments found. Creating new log.
I20260923 17:27:20.551510 26444 tablet_bootstrap.cc:654] T 02a47f27fd4f4079a5b14381752d0f85 P de8c684e115544cfa24e6fd295f49c9c: Neither blocks nor log segments found. Creating new log.
I20260923 17:27:20.551532 26445 log.cc:824] T 02a47f27fd4f4079a5b14381752d0f85 P 73d1d9b41a914836a80114437655ec6e: Log is configured to *not* fsync() on all Append() calls
I20260923 17:27:20.552721 26444 log.cc:824] T 02a47f27fd4f4079a5b14381752d0f85 P de8c684e115544cfa24e6fd295f49c9c: Log is configured to *not* fsync() on all Append() calls
I20260923 17:27:20.552991 26443 tablet_bootstrap.cc:492] T 02a47f27fd4f4079a5b14381752d0f85 P 300a1144ecb7434a9e1dd68677cab6cf: No bootstrap required, opened a new log
I20260923 17:27:20.553213 26443 ts_tablet_manager.cc:1397] T 02a47f27fd4f4079a5b14381752d0f85 P 300a1144ecb7434a9e1dd68677cab6cf: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260923 17:27:20.553397 26445 tablet_bootstrap.cc:492] T 02a47f27fd4f4079a5b14381752d0f85 P 73d1d9b41a914836a80114437655ec6e: No bootstrap required, opened a new log
I20260923 17:27:20.553601 26445 ts_tablet_manager.cc:1397] T 02a47f27fd4f4079a5b14381752d0f85 P 73d1d9b41a914836a80114437655ec6e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260923 17:27:20.554697 26444 tablet_bootstrap.cc:492] T 02a47f27fd4f4079a5b14381752d0f85 P de8c684e115544cfa24e6fd295f49c9c: No bootstrap required, opened a new log
I20260923 17:27:20.554888 26444 ts_tablet_manager.cc:1397] T 02a47f27fd4f4079a5b14381752d0f85 P de8c684e115544cfa24e6fd295f49c9c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 17:27:20.556519 26445 raft_consensus.cc:348] T 02a47f27fd4f4079a5b14381752d0f85 P 73d1d9b41a914836a80114437655ec6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d1d9b41a914836a80114437655ec6e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42075 } } peers { permanent_uuid: "de8c684e115544cfa24e6fd295f49c9c" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35255 } } peers { permanent_uuid: "300a1144ecb7434a9e1dd68677cab6cf" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 43095 } }
I20260923 17:27:20.556767 26445 raft_consensus.cc:374] T 02a47f27fd4f4079a5b14381752d0f85 P 73d1d9b41a914836a80114437655ec6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:27:20.556820 26445 raft_consensus.cc:729] T 02a47f27fd4f4079a5b14381752d0f85 P 73d1d9b41a914836a80114437655ec6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73d1d9b41a914836a80114437655ec6e, State: Initialized, Role: FOLLOWER
I20260923 17:27:20.557338 26445 consensus_queue.cc:260] T 02a47f27fd4f4079a5b14381752d0f85 P 73d1d9b41a914836a80114437655ec6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d1d9b41a914836a80114437655ec6e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42075 } } peers { permanent_uuid: "de8c684e115544cfa24e6fd295f49c9c" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35255 } } peers { permanent_uuid: "300a1144ecb7434a9e1dd68677cab6cf" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 43095 } }
I20260923 17:27:20.557981 26443 raft_consensus.cc:348] T 02a47f27fd4f4079a5b14381752d0f85 P 300a1144ecb7434a9e1dd68677cab6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d1d9b41a914836a80114437655ec6e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42075 } } peers { permanent_uuid: "de8c684e115544cfa24e6fd295f49c9c" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35255 } } peers { permanent_uuid: "300a1144ecb7434a9e1dd68677cab6cf" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 43095 } }
I20260923 17:27:20.558224 26443 raft_consensus.cc:374] T 02a47f27fd4f4079a5b14381752d0f85 P 300a1144ecb7434a9e1dd68677cab6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:27:20.558279 26443 raft_consensus.cc:729] T 02a47f27fd4f4079a5b14381752d0f85 P 300a1144ecb7434a9e1dd68677cab6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 300a1144ecb7434a9e1dd68677cab6cf, State: Initialized, Role: FOLLOWER
I20260923 17:27:20.558382 26445 ts_tablet_manager.cc:1428] T 02a47f27fd4f4079a5b14381752d0f85 P 73d1d9b41a914836a80114437655ec6e: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260923 17:27:20.558673 26443 consensus_queue.cc:260] T 02a47f27fd4f4079a5b14381752d0f85 P 300a1144ecb7434a9e1dd68677cab6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d1d9b41a914836a80114437655ec6e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42075 } } peers { permanent_uuid: "de8c684e115544cfa24e6fd295f49c9c" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35255 } } peers { permanent_uuid: "300a1144ecb7434a9e1dd68677cab6cf" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 43095 } }
I20260923 17:27:20.558660 26444 raft_consensus.cc:348] T 02a47f27fd4f4079a5b14381752d0f85 P de8c684e115544cfa24e6fd295f49c9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d1d9b41a914836a80114437655ec6e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42075 } } peers { permanent_uuid: "de8c684e115544cfa24e6fd295f49c9c" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35255 } } peers { permanent_uuid: "300a1144ecb7434a9e1dd68677cab6cf" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 43095 } }
I20260923 17:27:20.558812 26292 heartbeater.cc:503] Master 127.11.229.62:41427 was elected leader, sending a full tablet report...
I20260923 17:27:20.558876 26444 raft_consensus.cc:374] T 02a47f27fd4f4079a5b14381752d0f85 P de8c684e115544cfa24e6fd295f49c9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:27:20.558934 26444 raft_consensus.cc:729] T 02a47f27fd4f4079a5b14381752d0f85 P de8c684e115544cfa24e6fd295f49c9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de8c684e115544cfa24e6fd295f49c9c, State: Initialized, Role: FOLLOWER
I20260923 17:27:20.559338 26444 consensus_queue.cc:260] T 02a47f27fd4f4079a5b14381752d0f85 P de8c684e115544cfa24e6fd295f49c9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d1d9b41a914836a80114437655ec6e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42075 } } peers { permanent_uuid: "de8c684e115544cfa24e6fd295f49c9c" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35255 } } peers { permanent_uuid: "300a1144ecb7434a9e1dd68677cab6cf" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 43095 } }
I20260923 17:27:20.559397 26158 heartbeater.cc:503] Master 127.11.229.62:41427 was elected leader, sending a full tablet report...
I20260923 17:27:20.559648 26443 ts_tablet_manager.cc:1428] T 02a47f27fd4f4079a5b14381752d0f85 P 300a1144ecb7434a9e1dd68677cab6cf: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260923 17:27:20.560202 26444 ts_tablet_manager.cc:1428] T 02a47f27fd4f4079a5b14381752d0f85 P de8c684e115544cfa24e6fd295f49c9c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260923 17:27:20.560523 26426 heartbeater.cc:503] Master 127.11.229.62:41427 was elected leader, sending a full tablet report...
I20260923 17:27:20.623919 26449 raft_consensus.cc:482] T 02a47f27fd4f4079a5b14381752d0f85 P 73d1d9b41a914836a80114437655ec6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:27:20.624169 26449 raft_consensus.cc:504] T 02a47f27fd4f4079a5b14381752d0f85 P 73d1d9b41a914836a80114437655ec6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d1d9b41a914836a80114437655ec6e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42075 } } peers { permanent_uuid: "de8c684e115544cfa24e6fd295f49c9c" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35255 } } peers { permanent_uuid: "300a1144ecb7434a9e1dd68677cab6cf" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 43095 } }
I20260923 17:27:20.625694 26449 leader_election.cc:288] T 02a47f27fd4f4079a5b14381752d0f85 P 73d1d9b41a914836a80114437655ec6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de8c684e115544cfa24e6fd295f49c9c (127.11.229.3:35255), 300a1144ecb7434a9e1dd68677cab6cf (127.11.229.1:43095)
I20260923 17:27:20.629289 26294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:20.625891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:35255 (local address 127.11.229.2:59313)
0923 17:27:20.626086 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:20.626102 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:20.626196 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:20.626505 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:20.626507 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:20.626526 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:27:20.626646 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:20.626652 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:20.627396 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:20.627400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:20.628165 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:20.628173 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:20.628837 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:20.628841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:20.628951 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:20.628969 (+ 18us) client_negotiation.cc:769] Sending connection context
0923 17:27:20.629055 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 17:27:20.629134 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260923 17:27:20.629289 26441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:20.626015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:59313 (local address 127.11.229.3:35255)
0923 17:27:20.626156 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:20.626160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:20.626176 (+ 16us) server_negotiation.cc:407] Connection header received
0923 17:27:20.626345 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:20.626348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:20.626405 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:20.626491 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:27:20.626766 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:20.627278 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:20.628299 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:20.628708 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:20.628846 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:20.628898 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:20.629077 (+ 179us) server_negotiation.cc:299] Negotiation successful
0923 17:27:20.629145 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 17:27:20.629518 26442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:20.626162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:42159 (local address 127.11.229.1:43095)
0923 17:27:20.626314 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:20.626320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:20.626479 (+ 159us) server_negotiation.cc:407] Connection header received
0923 17:27:20.626638 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:20.626646 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:20.626689 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:20.626772 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:27:20.627045 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:20.627529 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:20.628536 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:20.628937 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:20.629068 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:20.629121 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:20.629341 (+ 220us) server_negotiation.cc:299] Negotiation successful
0923 17:27:20.629408 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 17:27:20.629530 26452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:20.625955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:43095 (local address 127.11.229.2:42159)
0923 17:27:20.626387 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:20.626398 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:20.626518 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:20.626794 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:20.626797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:20.626813 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:27:20.626935 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:20.626940 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:20.627636 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:20.627641 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:20.628410 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:20.628418 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:20.629056 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:20.629059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:20.629174 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:20.629200 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 17:27:20.629322 (+ 122us) client_negotiation.cc:240] Negotiation successful
0923 17:27:20.629393 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":120,"threads_started":1}
I20260923 17:27:20.630018 26362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02a47f27fd4f4079a5b14381752d0f85" candidate_uuid: "73d1d9b41a914836a80114437655ec6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de8c684e115544cfa24e6fd295f49c9c" is_pre_election: true
I20260923 17:27:20.630162 26112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02a47f27fd4f4079a5b14381752d0f85" candidate_uuid: "73d1d9b41a914836a80114437655ec6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "300a1144ecb7434a9e1dd68677cab6cf" is_pre_election: true
I20260923 17:27:20.630360 26362 raft_consensus.cc:2454] T 02a47f27fd4f4079a5b14381752d0f85 P de8c684e115544cfa24e6fd295f49c9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73d1d9b41a914836a80114437655ec6e in term 0.
I20260923 17:27:20.630403 26112 raft_consensus.cc:2454] T 02a47f27fd4f4079a5b14381752d0f85 P 300a1144ecb7434a9e1dd68677cab6cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73d1d9b41a914836a80114437655ec6e in term 0.
I20260923 17:27:20.630877 26181 leader_election.cc:302] T 02a47f27fd4f4079a5b14381752d0f85 P 73d1d9b41a914836a80114437655ec6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 300a1144ecb7434a9e1dd68677cab6cf, 73d1d9b41a914836a80114437655ec6e, de8c684e115544cfa24e6fd295f49c9c; no voters:
I20260923 17:27:20.631171 26449 raft_consensus.cc:2781] T 02a47f27fd4f4079a5b14381752d0f85 P 73d1d9b41a914836a80114437655ec6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:27:20.631258 26449 raft_consensus.cc:482] T 02a47f27fd4f4079a5b14381752d0f85 P 73d1d9b41a914836a80114437655ec6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:27:20.631311 26449 raft_consensus.cc:3037] T 02a47f27fd4f4079a5b14381752d0f85 P 73d1d9b41a914836a80114437655ec6e [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:27:20.632359 26449 raft_consensus.cc:504] T 02a47f27fd4f4079a5b14381752d0f85 P 73d1d9b41a914836a80114437655ec6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d1d9b41a914836a80114437655ec6e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42075 } } peers { permanent_uuid: "de8c684e115544cfa24e6fd295f49c9c" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35255 } } peers { permanent_uuid: "300a1144ecb7434a9e1dd68677cab6cf" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 43095 } }
I20260923 17:27:20.632701 26449 leader_election.cc:288] T 02a47f27fd4f4079a5b14381752d0f85 P 73d1d9b41a914836a80114437655ec6e [CANDIDATE]: Term 1 election: Requested vote from peers de8c684e115544cfa24e6fd295f49c9c (127.11.229.3:35255), 300a1144ecb7434a9e1dd68677cab6cf (127.11.229.1:43095)
I20260923 17:27:20.633076 26112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02a47f27fd4f4079a5b14381752d0f85" candidate_uuid: "73d1d9b41a914836a80114437655ec6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "300a1144ecb7434a9e1dd68677cab6cf"
I20260923 17:27:20.633074 26362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02a47f27fd4f4079a5b14381752d0f85" candidate_uuid: "73d1d9b41a914836a80114437655ec6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de8c684e115544cfa24e6fd295f49c9c"
I20260923 17:27:20.633200 26362 raft_consensus.cc:3037] T 02a47f27fd4f4079a5b14381752d0f85 P de8c684e115544cfa24e6fd295f49c9c [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:27:20.633201 26112 raft_consensus.cc:3037] T 02a47f27fd4f4079a5b14381752d0f85 P 300a1144ecb7434a9e1dd68677cab6cf [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:27:20.634511 26112 raft_consensus.cc:2454] T 02a47f27fd4f4079a5b14381752d0f85 P 300a1144ecb7434a9e1dd68677cab6cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73d1d9b41a914836a80114437655ec6e in term 1.
I20260923 17:27:20.634511 26362 raft_consensus.cc:2454] T 02a47f27fd4f4079a5b14381752d0f85 P de8c684e115544cfa24e6fd295f49c9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73d1d9b41a914836a80114437655ec6e in term 1.
I20260923 17:27:20.634796 26181 leader_election.cc:302] T 02a47f27fd4f4079a5b14381752d0f85 P 73d1d9b41a914836a80114437655ec6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73d1d9b41a914836a80114437655ec6e, de8c684e115544cfa24e6fd295f49c9c; no voters:
I20260923 17:27:20.634966 26449 raft_consensus.cc:2781] T 02a47f27fd4f4079a5b14381752d0f85 P 73d1d9b41a914836a80114437655ec6e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:27:20.635188 26449 raft_consensus.cc:686] T 02a47f27fd4f4079a5b14381752d0f85 P 73d1d9b41a914836a80114437655ec6e [term 1 LEADER]: Becoming Leader. State: Replica: 73d1d9b41a914836a80114437655ec6e, State: Running, Role: LEADER
I20260923 17:27:20.635527 26449 consensus_queue.cc:237] T 02a47f27fd4f4079a5b14381752d0f85 P 73d1d9b41a914836a80114437655ec6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d1d9b41a914836a80114437655ec6e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42075 } } peers { permanent_uuid: "de8c684e115544cfa24e6fd295f49c9c" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35255 } } peers { permanent_uuid: "300a1144ecb7434a9e1dd68677cab6cf" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 43095 } }
I20260923 17:27:20.638895 25963 catalog_manager.cc:5495] T 02a47f27fd4f4079a5b14381752d0f85 P 73d1d9b41a914836a80114437655ec6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 73d1d9b41a914836a80114437655ec6e (127.11.229.2). New cstate: current_term: 1 leader_uuid: "73d1d9b41a914836a80114437655ec6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d1d9b41a914836a80114437655ec6e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42075 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de8c684e115544cfa24e6fd295f49c9c" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "300a1144ecb7434a9e1dd68677cab6cf" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 43095 } health_report { overall_health: UNKNOWN } } }
I20260923 17:27:20.657438 26439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:20.653551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46956 (local address 127.11.229.2:42075)
0923 17:27:20.653723 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:20.653728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:20.653743 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:27:20.653858 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:20.653860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:20.653916 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:20.654027 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:27:20.654278 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:20.654791 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:20.655838 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:20.656074 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:20.656228 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:20.656549 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:27:20.656978 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:27:20.657121 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:20.657185 (+ 64us) server_negotiation.cc:299] Negotiation successful
0923 17:27:20.657245 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 17:27:20.663357 26112 raft_consensus.cc:1261] T 02a47f27fd4f4079a5b14381752d0f85 P 300a1144ecb7434a9e1dd68677cab6cf [term 1 FOLLOWER]: Refusing update from remote peer 73d1d9b41a914836a80114437655ec6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:27:20.663357 26362 raft_consensus.cc:1261] T 02a47f27fd4f4079a5b14381752d0f85 P de8c684e115544cfa24e6fd295f49c9c [term 1 FOLLOWER]: Refusing update from remote peer 73d1d9b41a914836a80114437655ec6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:27:20.664156 26453 consensus_queue.cc:1035] T 02a47f27fd4f4079a5b14381752d0f85 P 73d1d9b41a914836a80114437655ec6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "de8c684e115544cfa24e6fd295f49c9c" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:27:20.664355 26449 consensus_queue.cc:1035] T 02a47f27fd4f4079a5b14381752d0f85 P 73d1d9b41a914836a80114437655ec6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "300a1144ecb7434a9e1dd68677cab6cf" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 43095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:27:20.672429 26456 mvcc.cc:205] Tried to move back new op lower bound from 7332595468949872640 to 7332595468845907968. Current Snapshot: MvccSnapshot[applied={T|T < 7332595468949872640}]
I20260923 17:27:20.673878 26458 mvcc.cc:205] Tried to move back new op lower bound from 7332595468949872640 to 7332595468845907968. Current Snapshot: MvccSnapshot[applied={T|T < 7332595468949872640}]
I20260923 17:27:20.674088 26457 mvcc.cc:205] Tried to move back new op lower bound from 7332595468949872640 to 7332595468845907968. Current Snapshot: MvccSnapshot[applied={T|T < 7332595468949872640}]
I20260923 17:27:20.682386 25967 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:27:20.682571 25967 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:27:20.685305 25967 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311: Sending DeleteTablet for 3 replicas of tablet 02a47f27fd4f4079a5b14381752d0f85
I20260923 17:27:20.686079 26226 tablet_service.cc:1514] Processing DeleteTablet for tablet 02a47f27fd4f4079a5b14381752d0f85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:27:20 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:46950
I20260923 17:27:20.686152 26092 tablet_service.cc:1514] Processing DeleteTablet for tablet 02a47f27fd4f4079a5b14381752d0f85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:27:20 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:39960
I20260923 17:27:20.686127 26341 tablet_service.cc:1514] Processing DeleteTablet for tablet 02a47f27fd4f4079a5b14381752d0f85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:27:20 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:52264
I20260923 17:27:20.686627 26468 tablet_replica.cc:330] T 02a47f27fd4f4079a5b14381752d0f85 P de8c684e115544cfa24e6fd295f49c9c: stopping tablet replica
I20260923 17:27:20.686869 26467 tablet_replica.cc:330] T 02a47f27fd4f4079a5b14381752d0f85 P 300a1144ecb7434a9e1dd68677cab6cf: stopping tablet replica
I20260923 17:27:20.686959 26468 raft_consensus.cc:2229] T 02a47f27fd4f4079a5b14381752d0f85 P de8c684e115544cfa24e6fd295f49c9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:27:20.686983 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 26029
I20260923 17:27:20.687302 26468 raft_consensus.cc:2258] T 02a47f27fd4f4079a5b14381752d0f85 P de8c684e115544cfa24e6fd295f49c9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:27:20.688179 26466 tablet_replica.cc:330] T 02a47f27fd4f4079a5b14381752d0f85 P 73d1d9b41a914836a80114437655ec6e: stopping tablet replica
I20260923 17:27:20.688557 26466 raft_consensus.cc:2229] T 02a47f27fd4f4079a5b14381752d0f85 P 73d1d9b41a914836a80114437655ec6e [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:27:20.688956 26468 ts_tablet_manager.cc:1905] T 02a47f27fd4f4079a5b14381752d0f85 P de8c684e115544cfa24e6fd295f49c9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:27:20.689103 26466 raft_consensus.cc:2258] T 02a47f27fd4f4079a5b14381752d0f85 P 73d1d9b41a914836a80114437655ec6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:27:20.690618 26466 ts_tablet_manager.cc:1905] T 02a47f27fd4f4079a5b14381752d0f85 P 73d1d9b41a914836a80114437655ec6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:27:20.691978 26468 ts_tablet_manager.cc:1918] T 02a47f27fd4f4079a5b14381752d0f85 P de8c684e115544cfa24e6fd295f49c9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:27:20.692081 26468 log.cc:1196] T 02a47f27fd4f4079a5b14381752d0f85 P de8c684e115544cfa24e6fd295f49c9c: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/wal/wals/02a47f27fd4f4079a5b14381752d0f85
I20260923 17:27:20.692431 26468 ts_tablet_manager.cc:1939] T 02a47f27fd4f4079a5b14381752d0f85 P de8c684e115544cfa24e6fd295f49c9c: Deleting consensus metadata
I20260923 17:27:20.693837 25953 catalog_manager.cc:4841] TS de8c684e115544cfa24e6fd295f49c9c (127.11.229.3:35255): tablet 02a47f27fd4f4079a5b14381752d0f85 (table test-table [id=e57eb4a8d79c412b83814002504f0a39]) successfully deleted
I20260923 17:27:20.694248 26466 ts_tablet_manager.cc:1918] T 02a47f27fd4f4079a5b14381752d0f85 P 73d1d9b41a914836a80114437655ec6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:27:20.694350 26466 log.cc:1196] T 02a47f27fd4f4079a5b14381752d0f85 P 73d1d9b41a914836a80114437655ec6e: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/wal/wals/02a47f27fd4f4079a5b14381752d0f85
I20260923 17:27:20.694665 26466 ts_tablet_manager.cc:1939] T 02a47f27fd4f4079a5b14381752d0f85 P 73d1d9b41a914836a80114437655ec6e: Deleting consensus metadata
I20260923 17:27:20.695873 25954 catalog_manager.cc:4841] TS 73d1d9b41a914836a80114437655ec6e (127.11.229.2:42075): tablet 02a47f27fd4f4079a5b14381752d0f85 (table test-table [id=e57eb4a8d79c412b83814002504f0a39]) successfully deleted
W20260923 17:27:20.696414 25951 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.229.1:43095 (error 108)
I20260923 17:27:20.697500 26437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:20.697157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:43095 (local address 127.0.0.1:39962)
0923 17:27:20.697323 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:20.697399 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.1:43095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260923 17:27:20.697608 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 26163
W20260923 17:27:20.698426 25951 catalog_manager.cc:4584] TS 300a1144ecb7434a9e1dd68677cab6cf (127.11.229.1:43095): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 02a47f27fd4f4079a5b14381752d0f85: Network error: Client connection negotiation failed: client connection to 127.11.229.1:43095: connect: Connection refused (error 111)
I20260923 17:27:20.703912 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 26297
I20260923 17:27:20.710012 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 25935
I20260923 17:27:20.716915 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:41427
--webserver_interface=127.11.229.62
--webserver_port=43819
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:44585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:41427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:27:20.830163 26470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:20.830422 26470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:20.830461 26470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:20.834508 26470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:27:20.834563 26470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:20.834583 26470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:20.834599 26470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:27:20.834615 26470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:27:20.839864 26470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:44585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:41427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:41427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=43819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:27:20.840883 26470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:20.841935 26470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:20.856714 26476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:20.856812 26470 server_base.cc:1034] running on GCE node
W20260923 17:27:20.856683 26479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:20.856707 26477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:20.857658 26470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:20.858878 26470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:20.860072 26470 hybrid_clock.cc:648] HybridClock initialized: now 1790184440860054 us; error 31 us; skew 500 ppm
Sep 23 17:27:20 dist-test-slave-ptl9 krb5kdc[25922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184440, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:20.862850 26470 init.cc:377] Logged in from keytab as kudu/127.11.229.62@KRBTEST.COM (short username kudu)
I20260923 17:27:20.863837 26470 webserver.cc:460] Webserver started at http://127.11.229.62:43819/ using document root <none> and password file <none>
I20260923 17:27:20.864462 26470 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:20.864578 26470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:20.868629 26470 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:27:20.870982 26486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:20.872279 26470 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:27:20.872433 26470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "c863213e6d3941c2890d8e505df89311"
format_stamp: "Formatted at 2026-09-23 17:27:19 on dist-test-slave-ptl9"
server_key: "f145fb9a8be9e7b499670d19cf3e76c7"
server_key_iv: "2e902bb5d573604c3c2ee4cff81d027e"
server_key_version: "encryptionkey@0"
I20260923 17:27:20.873087 26470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:20.898538 26470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:20.901640 26470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:20.901853 26470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:20.909816 26470 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:41427
I20260923 17:27:20.910149 26538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:41427 every 8 connection(s)
I20260923 17:27:20.911145 26470 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:27:20.914140 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 26470
I20260923 17:27:20.914587 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.1:43095
--local_ip_for_outbound_sockets=127.11.229.1
--tserver_master_addrs=127.11.229.62:41427
--webserver_port=34821
--webserver_interface=127.11.229.1
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:44585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:27:20.916072 26539 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:27:20.917909 26539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311: Bootstrap starting.
I20260923 17:27:20.935501 26539 log.cc:824] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311: Log is configured to *not* fsync() on all Append() calls
I20260923 17:27:20.944602 26539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:27:20.945007 26539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311: Bootstrap complete.
I20260923 17:27:20.949497 26539 raft_consensus.cc:348] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c863213e6d3941c2890d8e505df89311" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 41427 } }
I20260923 17:27:20.949957 26539 raft_consensus.cc:729] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c863213e6d3941c2890d8e505df89311, State: Initialized, Role: FOLLOWER
I20260923 17:27:20.950460 26539 consensus_queue.cc:260] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c863213e6d3941c2890d8e505df89311" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 41427 } }
I20260923 17:27:20.950606 26539 raft_consensus.cc:388] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:27:20.950704 26539 raft_consensus.cc:482] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:27:20.950829 26539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [term 1 FOLLOWER]: Advancing to term 2
I20260923 17:27:20.952785 26539 raft_consensus.cc:504] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c863213e6d3941c2890d8e505df89311" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 41427 } }
I20260923 17:27:20.953145 26539 leader_election.cc:302] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c863213e6d3941c2890d8e505df89311; no voters:
I20260923 17:27:20.953548 26539 leader_election.cc:288] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 17:27:20.953685 26543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 17:27:20.953914 26543 raft_consensus.cc:686] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [term 2 LEADER]: Becoming Leader. State: Replica: c863213e6d3941c2890d8e505df89311, State: Running, Role: LEADER
I20260923 17:27:20.954205 26543 consensus_queue.cc:237] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c863213e6d3941c2890d8e505df89311" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 41427 } }
I20260923 17:27:20.954818 26539 sys_catalog.cc:564] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:27:20.955197 26545 sys_catalog.cc:455] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c863213e6d3941c2890d8e505df89311. Latest consensus state: current_term: 2 leader_uuid: "c863213e6d3941c2890d8e505df89311" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c863213e6d3941c2890d8e505df89311" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 41427 } } }
I20260923 17:27:20.955327 26545 sys_catalog.cc:458] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:20.955526 26544 sys_catalog.cc:455] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c863213e6d3941c2890d8e505df89311" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c863213e6d3941c2890d8e505df89311" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 41427 } } }
I20260923 17:27:20.955631 26544 sys_catalog.cc:458] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:20.955688 26547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:27:20.959911 26547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:27:20.960595 26547 catalog_manager.cc:1259] Loaded cluster ID: 66a085f4ceb9403bbb8e345b2a1ad23f
I20260923 17:27:20.960666 26547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:27:20.961840 26547 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260923 17:27:20.964546 26547 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260923 17:27:20.968698 26547 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:27:20.969476 26547 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311: Loaded TSK: 0
I20260923 17:27:20.970415 26547 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 17:27:21.037184 26541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:21.037556 26541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:21.037642 26541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:21.041775 26541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:21.041864 26541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:21.041958 26541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.1
I20260923 17:27:21.047322 26541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:44585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.1:43095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.229.1
--webserver_port=34821
--tserver_master_addrs=127.11.229.62:41427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.1
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:27:21.048430 26541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:21.049520 26541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:21.060438 26569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:21.060449 26567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:21.060696 26541 server_base.cc:1034] running on GCE node
W20260923 17:27:21.060432 26566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:21.061368 26541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:21.062013 26541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:21.063205 26541 hybrid_clock.cc:648] HybridClock initialized: now 1790184441063180 us; error 46 us; skew 500 ppm
Sep 23 17:27:21 dist-test-slave-ptl9 krb5kdc[25922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184441, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:21.065948 26541 init.cc:377] Logged in from keytab as kudu/127.11.229.1@KRBTEST.COM (short username kudu)
I20260923 17:27:21.066884 26541 webserver.cc:460] Webserver started at http://127.11.229.1:34821/ using document root <none> and password file <none>
I20260923 17:27:21.067467 26541 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:21.067543 26541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:21.071683 26541 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:27:21.074002 26576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:21.075357 26541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:27:21.075500 26541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/wal
uuid: "300a1144ecb7434a9e1dd68677cab6cf"
format_stamp: "Formatted at 2026-09-23 17:27:19 on dist-test-slave-ptl9"
server_key: "3cae6515633ab0f53fe1de8dace8a170"
server_key_iv: "975ef45b8786364365e90e24a6b7f52c"
server_key_version: "encryptionkey@0"
I20260923 17:27:21.076136 26541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:21.087138 26541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:21.090461 26541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:21.090668 26541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:21.091346 26541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:27:21.092533 26583 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 17:27:21.096804 26541 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 17:27:21.096868 26541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260923 17:27:21.096936 26541 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 17:27:21.098706 26541 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 17:27:21.098762 26541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260923 17:27:21.099037 26583 tablet_bootstrap.cc:492] T 02a47f27fd4f4079a5b14381752d0f85 P 300a1144ecb7434a9e1dd68677cab6cf: Bootstrap starting.
I20260923 17:27:21.109807 26541 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.1:43095
I20260923 17:27:21.111064 26541 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
I20260923 17:27:21.111429 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 26541
I20260923 17:27:21.111888 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.2:42075
--local_ip_for_outbound_sockets=127.11.229.2
--tserver_master_addrs=127.11.229.62:41427
--webserver_port=35055
--webserver_interface=127.11.229.2
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:44585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:27:21.115631 26690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.1:43095 every 8 connection(s)
I20260923 17:27:21.120285 26583 log.cc:824] T 02a47f27fd4f4079a5b14381752d0f85 P 300a1144ecb7434a9e1dd68677cab6cf: Log is configured to *not* fsync() on all Append() calls
Sep 23 17:27:21 dist-test-slave-ptl9 krb5kdc[25922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184441, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:27:21.128832 26583 tablet_bootstrap.cc:492] T 02a47f27fd4f4079a5b14381752d0f85 P 300a1144ecb7434a9e1dd68677cab6cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:27:21.129303 26583 tablet_bootstrap.cc:492] T 02a47f27fd4f4079a5b14381752d0f85 P 300a1144ecb7434a9e1dd68677cab6cf: Bootstrap complete.
I20260923 17:27:21.130650 26583 ts_tablet_manager.cc:1397] T 02a47f27fd4f4079a5b14381752d0f85 P 300a1144ecb7434a9e1dd68677cab6cf: Time spent bootstrapping tablet: real 0.032s user 0.025s sys 0.005s
I20260923 17:27:21.132866 26695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:21.114256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.1:34925)
0923 17:27:21.116080 (+ 1824us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:21.116134 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:21.117069 (+ 935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:21.118301 (+ 1232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:21.118310 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:21.118749 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:21.119234 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:21.119249 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.120631 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.120638 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:21.122034 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:21.122047 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.123298 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.123302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:21.123493 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:21.124233 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:21.124254 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:21.126347 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:21.129296 (+ 2949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:21.129302 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:21.129318 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:21.129680 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:21.130146 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:21.130152 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:21.130157 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:21.130348 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:21.131061 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:21.131068 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:21.131365 (+ 297us) client_negotiation.cc:769] Sending connection context
0923 17:27:21.131561 (+ 196us) client_negotiation.cc:240] Negotiation successful
0923 17:27:21.131896 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1626,"thread_start_us":144,"threads_started":1}
I20260923 17:27:21.132956 26694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:21.114145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:34925 (local address 127.11.229.62:41427)
0923 17:27:21.114493 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:21.114510 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:21.116273 (+ 1763us) server_negotiation.cc:407] Connection header received
0923 17:27:21.117571 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:21.117583 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:21.117897 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:21.118245 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:21.119433 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.120475 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.122202 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.123103 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.123321 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:21.126610 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:21.126621 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:21.126645 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:21.126690 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:21.129125 (+ 2435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:21.129838 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:21.129847 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:21.129852 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:21.130031 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:21.130465 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:21.130469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:21.130471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:21.130938 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:21.131070 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:21.131720 (+ 650us) server_negotiation.cc:299] Negotiation successful
0923 17:27:21.132123 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":119,"threads_started":1}
I20260923 17:27:21.133736 26583 raft_consensus.cc:348] T 02a47f27fd4f4079a5b14381752d0f85 P 300a1144ecb7434a9e1dd68677cab6cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d1d9b41a914836a80114437655ec6e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42075 } } peers { permanent_uuid: "de8c684e115544cfa24e6fd295f49c9c" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35255 } } peers { permanent_uuid: "300a1144ecb7434a9e1dd68677cab6cf" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 43095 } }
I20260923 17:27:21.134271 26583 raft_consensus.cc:729] T 02a47f27fd4f4079a5b14381752d0f85 P 300a1144ecb7434a9e1dd68677cab6cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 300a1144ecb7434a9e1dd68677cab6cf, State: Initialized, Role: FOLLOWER
I20260923 17:27:21.134773 26583 consensus_queue.cc:260] T 02a47f27fd4f4079a5b14381752d0f85 P 300a1144ecb7434a9e1dd68677cab6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d1d9b41a914836a80114437655ec6e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42075 } } peers { permanent_uuid: "de8c684e115544cfa24e6fd295f49c9c" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35255 } } peers { permanent_uuid: "300a1144ecb7434a9e1dd68677cab6cf" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 43095 } }
I20260923 17:27:21.135874 26583 ts_tablet_manager.cc:1428] T 02a47f27fd4f4079a5b14381752d0f85 P 300a1144ecb7434a9e1dd68677cab6cf: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260923 17:27:21.136027 26691 heartbeater.cc:347] Connected to a master server at 127.11.229.62:41427
I20260923 17:27:21.136260 26691 heartbeater.cc:464] Registering TS with master...
I20260923 17:27:21.136832 26691 heartbeater.cc:511] Master 127.11.229.62:41427 requested a full tablet report, sending...
I20260923 17:27:21.139622 26503 ts_manager.cc:194] Registered new tserver with Master: 300a1144ecb7434a9e1dd68677cab6cf (127.11.229.1:43095)
I20260923 17:27:21.144022 26503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.1@KRBTEST.COM'} at 127.11.229.1:34925
I20260923 17:27:21.146772 26691 heartbeater.cc:503] Master 127.11.229.62:41427 was elected leader, sending a full tablet report...
Sep 23 17:27:21 dist-test-slave-ptl9 krb5kdc[25922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184440, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for kudu/127.11.229.1@KRBTEST.COM
I20260923 17:27:21.153348 26698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:21.142980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:43095 (local address 127.0.0.1:39976)
0923 17:27:21.143700 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:21.143744 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:21.143909 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:21.144438 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:21.144444 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:21.144878 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:21.144995 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:21.145002 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.146061 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.146066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:21.146700 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:21.146717 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.147434 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.147438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:21.147558 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:21.148138 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:21.148150 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:21.149634 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:21.151450 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:21.151460 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:21.151466 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:21.151721 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:21.152038 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:21.152040 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:21.152042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:21.152129 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:21.152640 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:21.152646 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:21.152778 (+ 132us) client_negotiation.cc:769] Sending connection context
0923 17:27:21.152917 (+ 139us) client_negotiation.cc:240] Negotiation successful
0923 17:27:21.153126 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":112,"threads_started":1}
I20260923 17:27:21.153671 26699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:21.143047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39976 (local address 127.11.229.1:43095)
0923 17:27:21.143461 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:21.143472 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:21.143839 (+ 367us) server_negotiation.cc:407] Connection header received
0923 17:27:21.144063 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:21.144068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:21.144292 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:21.144438 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:21.145112 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.145948 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.146832 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.147311 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.147418 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:21.149782 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:21.149791 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:21.149804 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:21.149836 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:21.151336 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:21.151840 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:21.151846 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:21.151848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:21.151935 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:21.152225 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:21.152228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:21.152230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:21.152527 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:21.152667 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:21.152997 (+ 330us) server_negotiation.cc:299] Negotiation successful
0923 17:27:21.153198 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":10,"server-negotiator.queue_time_us":308,"thread_start_us":104,"threads_started":1}
I20260923 17:27:21.154560 26625 tablet_service.cc:1514] Processing DeleteTablet for tablet 02a47f27fd4f4079a5b14381752d0f85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:27:20 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:39976
I20260923 17:27:21.155532 26700 tablet_replica.cc:330] stopping tablet replica
I20260923 17:27:21.155879 26700 raft_consensus.cc:2229] T 02a47f27fd4f4079a5b14381752d0f85 P 300a1144ecb7434a9e1dd68677cab6cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:27:21.156122 26700 raft_consensus.cc:2258] T 02a47f27fd4f4079a5b14381752d0f85 P 300a1144ecb7434a9e1dd68677cab6cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:27:21.157101 26700 ts_tablet_manager.cc:1905] T 02a47f27fd4f4079a5b14381752d0f85 P 300a1144ecb7434a9e1dd68677cab6cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:27:21.160717 26700 ts_tablet_manager.cc:1918] T 02a47f27fd4f4079a5b14381752d0f85 P 300a1144ecb7434a9e1dd68677cab6cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:27:21.160833 26700 log.cc:1196] T 02a47f27fd4f4079a5b14381752d0f85 P 300a1144ecb7434a9e1dd68677cab6cf: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/wal/wals/02a47f27fd4f4079a5b14381752d0f85
I20260923 17:27:21.161089 26700 ts_tablet_manager.cc:1939] T 02a47f27fd4f4079a5b14381752d0f85 P 300a1144ecb7434a9e1dd68677cab6cf: Deleting consensus metadata
I20260923 17:27:21.162776 26487 catalog_manager.cc:4841] TS 300a1144ecb7434a9e1dd68677cab6cf (127.11.229.1:43095): tablet 02a47f27fd4f4079a5b14381752d0f85 (table test-table [id=e57eb4a8d79c412b83814002504f0a39]) successfully deleted
W20260923 17:27:21.239859 26693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:21.240108 26693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:21.240150 26693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:21.244120 26693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:21.244176 26693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:21.244280 26693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.2
I20260923 17:27:21.249423 26693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:44585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.2:42075
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.229.2
--webserver_port=35055
--tserver_master_addrs=127.11.229.62:41427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.2
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:27:21.250453 26693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:21.251482 26693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:21.261142 26705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:21.261157 26708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:21.261281 26693 server_base.cc:1034] running on GCE node
W20260923 17:27:21.261137 26706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:21.261829 26693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:21.262446 26693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:21.263603 26693 hybrid_clock.cc:648] HybridClock initialized: now 1790184441263569 us; error 45 us; skew 500 ppm
Sep 23 17:27:21 dist-test-slave-ptl9 krb5kdc[25922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184441, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:21.266374 26693 init.cc:377] Logged in from keytab as kudu/127.11.229.2@KRBTEST.COM (short username kudu)
I20260923 17:27:21.267280 26693 webserver.cc:460] Webserver started at http://127.11.229.2:35055/ using document root <none> and password file <none>
I20260923 17:27:21.267803 26693 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:21.267872 26693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:21.271679 26693 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:27:21.273970 26715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:21.275239 26693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:27:21.275334 26693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/wal
uuid: "73d1d9b41a914836a80114437655ec6e"
format_stamp: "Formatted at 2026-09-23 17:27:20 on dist-test-slave-ptl9"
server_key: "4431f6672aa0ddca05bc42c366860378"
server_key_iv: "beae22808722dc86821f544c122d1c4b"
server_key_version: "encryptionkey@0"
I20260923 17:27:21.275853 26693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:21.287461 26693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:21.290647 26693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:21.290792 26693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:21.291289 26693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:27:21.292052 26693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:27:21.292096 26693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:21.292135 26693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:27:21.292150 26693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:21.301215 26693 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.2:42075
I20260923 17:27:21.301306 26828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.2:42075 every 8 connection(s)
I20260923 17:27:21.302301 26693 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
I20260923 17:27:21.304848 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 26693
I20260923 17:27:21.305359 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.3:35255
--local_ip_for_outbound_sockets=127.11.229.3
--tserver_master_addrs=127.11.229.62:41427
--webserver_port=42641
--webserver_interface=127.11.229.3
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:44585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:27:21 dist-test-slave-ptl9 krb5kdc[25922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184441, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:27:21.317444 26694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:21.304262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:53367 (local address 127.11.229.62:41427)
0923 17:27:21.304412 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:21.304417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:21.305528 (+ 1111us) server_negotiation.cc:407] Connection header received
0923 17:27:21.307291 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:21.307295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:21.307361 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:21.307468 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:21.308863 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.309471 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.310649 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.310886 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.311001 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:21.313834 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:21.313838 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:21.313840 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:21.313865 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:21.315320 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:21.315829 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:21.315835 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:21.315837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:21.315923 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:21.316307 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:21.316312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:21.316314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:21.316501 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:21.316621 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:21.317115 (+ 494us) server_negotiation.cc:299] Negotiation successful
0923 17:27:21.317251 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:27:21.318325 26831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:21.304720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.2:53367)
0923 17:27:21.305357 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:21.305403 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:21.307063 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:21.307684 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:21.307693 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:21.308164 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:21.308713 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:21.308727 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.309604 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.309608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:21.310515 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:21.310525 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.311044 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.311048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:21.311231 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:21.311890 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:21.311907 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:21.313653 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:21.315441 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:21.315446 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:21.315462 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:21.315724 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:21.316017 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:21.316021 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:21.316023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:21.316206 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:21.316609 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:21.316616 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:21.316867 (+ 251us) client_negotiation.cc:769] Sending connection context
0923 17:27:21.317059 (+ 192us) client_negotiation.cc:240] Negotiation successful
0923 17:27:21.317430 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":156,"threads_started":1}
I20260923 17:27:21.319672 26829 heartbeater.cc:347] Connected to a master server at 127.11.229.62:41427
I20260923 17:27:21.319934 26829 heartbeater.cc:464] Registering TS with master...
I20260923 17:27:21.320446 26829 heartbeater.cc:511] Master 127.11.229.62:41427 requested a full tablet report, sending...
I20260923 17:27:21.321617 26503 ts_manager.cc:194] Registered new tserver with Master: 73d1d9b41a914836a80114437655ec6e (127.11.229.2:42075)
I20260923 17:27:21.322431 26503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.2@KRBTEST.COM'} at 127.11.229.2:53367
W20260923 17:27:21.421443 26832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:21.421693 26832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:21.421736 26832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:21.425664 26832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:21.425715 26832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:21.425771 26832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.3
I20260923 17:27:21.430799 26832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:44585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.3:35255
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.229.3
--webserver_port=42641
--tserver_master_addrs=127.11.229.62:41427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.3
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:27:21.431843 26832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:21.432850 26832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:21.442484 26837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:21.442480 26838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:21.442480 26840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:21.443014 26832 server_base.cc:1034] running on GCE node
I20260923 17:27:21.443384 26832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:21.443949 26832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:21.445137 26832 hybrid_clock.cc:648] HybridClock initialized: now 1790184441445117 us; error 31 us; skew 500 ppm
Sep 23 17:27:21 dist-test-slave-ptl9 krb5kdc[25922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184441, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:21.447981 26832 init.cc:377] Logged in from keytab as kudu/127.11.229.3@KRBTEST.COM (short username kudu)
I20260923 17:27:21.448886 26832 webserver.cc:460] Webserver started at http://127.11.229.3:42641/ using document root <none> and password file <none>
I20260923 17:27:21.449522 26832 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:21.449599 26832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:21.453584 26832 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:27:21.455717 26847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:21.456986 26832 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:27:21.457180 26832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/wal
uuid: "de8c684e115544cfa24e6fd295f49c9c"
format_stamp: "Formatted at 2026-09-23 17:27:20 on dist-test-slave-ptl9"
server_key: "3baced430a71467226be102e91ac1056"
server_key_iv: "99e00b927367ff9332dedf682ce6f80e"
server_key_version: "encryptionkey@0"
I20260923 17:27:21.457763 26832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:21.476032 26832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:21.479573 26832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:21.479787 26832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:21.480407 26832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:27:21.481345 26832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:27:21.481393 26832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:21.481436 26832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:27:21.481456 26832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:21.490303 26832 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.3:35255
I20260923 17:27:21.490350 26960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.3:35255 every 8 connection(s)
I20260923 17:27:21.491369 26832 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
I20260923 17:27:21.492601 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 26832
I20260923 17:27:21.501744 26694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:21.493377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41462 (local address 127.11.229.62:41427)
0923 17:27:21.493528 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:21.493533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:21.493737 (+ 204us) server_negotiation.cc:407] Connection header received
0923 17:27:21.493917 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:21.493921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:21.493975 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:21.494071 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:21.494801 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.495344 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.496376 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.496719 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.496856 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:21.498381 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:21.498385 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:21.498387 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:21.498410 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:21.499982 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:21.500441 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:21.500444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:21.500445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:21.500521 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:21.500818 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:21.500822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:21.500823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:21.501037 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:21.501248 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:21.501451 (+ 203us) server_negotiation.cc:299] Negotiation successful
0923 17:27:21.501575 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Sep 23 17:27:21 dist-test-slave-ptl9 krb5kdc[25922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184441, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:27:21.506970 26964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:21.493663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:51493 (local address 127.11.229.62:41427)
0923 17:27:21.493898 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:21.493904 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:21.494694 (+ 790us) server_negotiation.cc:407] Connection header received
0923 17:27:21.496149 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:21.496152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:21.496205 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:21.496311 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:21.497809 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.498566 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.500324 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.500651 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.500768 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:21.503604 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:21.503607 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:21.503610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:21.503637 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:21.504989 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:21.505555 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:21.505557 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:21.505558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:21.505634 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:21.505939 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:21.505941 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:21.505942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:21.506180 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:21.506290 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:21.506741 (+ 451us) server_negotiation.cc:299] Negotiation successful
0923 17:27:21.506843 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":66,"threads_started":1}
I20260923 17:27:21.507855 26965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:21.494000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.3:51493)
0923 17:27:21.494533 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:21.494585 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:21.495949 (+ 1364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:21.496496 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:21.496504 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:21.497008 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:21.497659 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:21.497676 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.498703 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.498705 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:21.500203 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:21.500212 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.500827 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.500832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:21.501032 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:21.501693 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:21.501715 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:21.503442 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:21.505093 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:21.505098 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:21.505111 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:21.505460 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:21.505720 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:21.505722 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:21.505724 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:21.505853 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:21.506284 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:21.506290 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:21.506524 (+ 234us) client_negotiation.cc:769] Sending connection context
0923 17:27:21.506678 (+ 154us) client_negotiation.cc:240] Negotiation successful
0923 17:27:21.506942 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":169,"threads_started":1}
I20260923 17:27:21.509106 26961 heartbeater.cc:347] Connected to a master server at 127.11.229.62:41427
I20260923 17:27:21.509413 26961 heartbeater.cc:464] Registering TS with master...
I20260923 17:27:21.509920 26961 heartbeater.cc:511] Master 127.11.229.62:41427 requested a full tablet report, sending...
I20260923 17:27:21.511000 26503 ts_manager.cc:194] Registered new tserver with Master: de8c684e115544cfa24e6fd295f49c9c (127.11.229.3:35255)
I20260923 17:27:21.511574 26503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.3@KRBTEST.COM'} at 127.11.229.3:51493
I20260923 17:27:21.515561 12180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:27:21.519706 26964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:21.516148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41476 (local address 127.11.229.62:41427)
0923 17:27:21.516295 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:21.516301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:21.516492 (+ 191us) server_negotiation.cc:407] Connection header received
0923 17:27:21.516650 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:21.516652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:21.516752 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:21.516835 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:27:21.517040 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.517616 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.518441 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.518655 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.518730 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:21.518987 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:27:21.519346 (+ 359us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:27:21.519433 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:21.519490 (+ 57us) server_negotiation.cc:299] Negotiation successful
0923 17:27:21.519536 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 17:27:21.520318 26503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:41476:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:27:21.523677 26503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:27:21.531432 26625 tablet_service.cc:1467] Processing CreateTablet for tablet ea02f341fd68421d8a5b0e7b54b86bf5 (DEFAULT_TABLE table=test-table [id=f3d81d167124402298a8d0e83e84b789]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:27:21.532123 26625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea02f341fd68421d8a5b0e7b54b86bf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:27:21.534989 26583 tablet_bootstrap.cc:492] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf: Bootstrap starting.
I20260923 17:27:21.535903 26583 tablet_bootstrap.cc:654] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf: Neither blocks nor log segments found. Creating new log.
I20260923 17:27:21.536368 26698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:21.530408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:35255 (local address 127.0.0.1:52278)
0923 17:27:21.530525 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:21.530542 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:21.530621 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:21.531553 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:21.531556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:21.531574 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:27:21.531685 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:21.531691 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.533602 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.533604 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:21.534900 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:21.534913 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.535809 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.535816 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:21.535944 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:21.535997 (+ 53us) client_negotiation.cc:769] Sending connection context
0923 17:27:21.536089 (+ 92us) client_negotiation.cc:240] Negotiation successful
0923 17:27:21.536184 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":51}
I20260923 17:27:21.536577 26970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:21.530629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52278 (local address 127.11.229.3:35255)
0923 17:27:21.531033 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:21.531046 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:21.531070 (+ 24us) server_negotiation.cc:407] Connection header received
0923 17:27:21.531148 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:21.531153 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:21.531401 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:21.531580 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:27:21.531862 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.533428 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.535046 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.535669 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.535760 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:21.535872 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:21.536312 (+ 440us) server_negotiation.cc:299] Negotiation successful
0923 17:27:21.536416 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":161,"threads_started":1}
I20260923 17:27:21.537653 26583 tablet_bootstrap.cc:492] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf: No bootstrap required, opened a new log
I20260923 17:27:21.537770 26583 ts_tablet_manager.cc:1397] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 17:27:21.537840 26971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:21.530853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:42075 (local address 127.0.0.1:46958)
0923 17:27:21.531239 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:21.531252 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:21.531347 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:21.532057 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:21.532060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:21.532078 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:27:21.532168 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:21.532174 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.533520 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.533526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:21.536129 (+ 2603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:21.536137 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.537451 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.537455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:21.537561 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:21.537582 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 17:27:21.537632 (+ 50us) client_negotiation.cc:240] Negotiation successful
0923 17:27:21.537688 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":1241,"thread_start_us":85,"threads_started":1}
I20260923 17:27:21.538240 26583 raft_consensus.cc:348] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de8c684e115544cfa24e6fd295f49c9c" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35255 } } peers { permanent_uuid: "73d1d9b41a914836a80114437655ec6e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42075 } } peers { permanent_uuid: "300a1144ecb7434a9e1dd68677cab6cf" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 43095 } }
I20260923 17:27:21.538367 26583 raft_consensus.cc:374] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:27:21.538420 26583 raft_consensus.cc:729] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 300a1144ecb7434a9e1dd68677cab6cf, State: Initialized, Role: FOLLOWER
I20260923 17:27:21.538439 26972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:21.531175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46958 (local address 127.11.229.2:42075)
0923 17:27:21.531584 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:21.531595 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:21.531616 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:27:21.531687 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:21.531692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:21.531896 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:21.532079 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:27:21.532305 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.533227 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.536249 (+ 3022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.537303 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.538052 (+ 749us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:21.538186 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:21.538229 (+ 43us) server_negotiation.cc:299] Negotiation successful
0923 17:27:21.538296 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":151,"threads_started":1}
I20260923 17:27:21.540493 26583 consensus_queue.cc:260] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de8c684e115544cfa24e6fd295f49c9c" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35255 } } peers { permanent_uuid: "73d1d9b41a914836a80114437655ec6e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42075 } } peers { permanent_uuid: "300a1144ecb7434a9e1dd68677cab6cf" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 43095 } }
I20260923 17:27:21.540793 26583 ts_tablet_manager.cc:1428] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260923 17:27:21.540585 26895 tablet_service.cc:1467] Processing CreateTablet for tablet ea02f341fd68421d8a5b0e7b54b86bf5 (DEFAULT_TABLE table=test-table [id=f3d81d167124402298a8d0e83e84b789]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:27:21.541831 26895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea02f341fd68421d8a5b0e7b54b86bf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:27:21.543053 26763 tablet_service.cc:1467] Processing CreateTablet for tablet ea02f341fd68421d8a5b0e7b54b86bf5 (DEFAULT_TABLE table=test-table [id=f3d81d167124402298a8d0e83e84b789]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:27:21.544201 26763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea02f341fd68421d8a5b0e7b54b86bf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:27:21.550277 26974 tablet_bootstrap.cc:492] T ea02f341fd68421d8a5b0e7b54b86bf5 P de8c684e115544cfa24e6fd295f49c9c: Bootstrap starting.
I20260923 17:27:21.552296 26975 tablet_bootstrap.cc:492] T ea02f341fd68421d8a5b0e7b54b86bf5 P 73d1d9b41a914836a80114437655ec6e: Bootstrap starting.
I20260923 17:27:21.552404 26974 tablet_bootstrap.cc:654] T ea02f341fd68421d8a5b0e7b54b86bf5 P de8c684e115544cfa24e6fd295f49c9c: Neither blocks nor log segments found. Creating new log.
I20260923 17:27:21.553833 26974 log.cc:824] T ea02f341fd68421d8a5b0e7b54b86bf5 P de8c684e115544cfa24e6fd295f49c9c: Log is configured to *not* fsync() on all Append() calls
I20260923 17:27:21.555416 26975 tablet_bootstrap.cc:654] T ea02f341fd68421d8a5b0e7b54b86bf5 P 73d1d9b41a914836a80114437655ec6e: Neither blocks nor log segments found. Creating new log.
I20260923 17:27:21.556190 26974 tablet_bootstrap.cc:492] T ea02f341fd68421d8a5b0e7b54b86bf5 P de8c684e115544cfa24e6fd295f49c9c: No bootstrap required, opened a new log
I20260923 17:27:21.556351 26975 log.cc:824] T ea02f341fd68421d8a5b0e7b54b86bf5 P 73d1d9b41a914836a80114437655ec6e: Log is configured to *not* fsync() on all Append() calls
I20260923 17:27:21.556413 26974 ts_tablet_manager.cc:1397] T ea02f341fd68421d8a5b0e7b54b86bf5 P de8c684e115544cfa24e6fd295f49c9c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260923 17:27:21.558368 26975 tablet_bootstrap.cc:492] T ea02f341fd68421d8a5b0e7b54b86bf5 P 73d1d9b41a914836a80114437655ec6e: No bootstrap required, opened a new log
I20260923 17:27:21.558560 26975 ts_tablet_manager.cc:1397] T ea02f341fd68421d8a5b0e7b54b86bf5 P 73d1d9b41a914836a80114437655ec6e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260923 17:27:21.560364 26974 raft_consensus.cc:348] T ea02f341fd68421d8a5b0e7b54b86bf5 P de8c684e115544cfa24e6fd295f49c9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de8c684e115544cfa24e6fd295f49c9c" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35255 } } peers { permanent_uuid: "73d1d9b41a914836a80114437655ec6e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42075 } } peers { permanent_uuid: "300a1144ecb7434a9e1dd68677cab6cf" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 43095 } }
I20260923 17:27:21.560649 26974 raft_consensus.cc:374] T ea02f341fd68421d8a5b0e7b54b86bf5 P de8c684e115544cfa24e6fd295f49c9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:27:21.560714 26974 raft_consensus.cc:729] T ea02f341fd68421d8a5b0e7b54b86bf5 P de8c684e115544cfa24e6fd295f49c9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de8c684e115544cfa24e6fd295f49c9c, State: Initialized, Role: FOLLOWER
I20260923 17:27:21.561173 26974 consensus_queue.cc:260] T ea02f341fd68421d8a5b0e7b54b86bf5 P de8c684e115544cfa24e6fd295f49c9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de8c684e115544cfa24e6fd295f49c9c" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35255 } } peers { permanent_uuid: "73d1d9b41a914836a80114437655ec6e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42075 } } peers { permanent_uuid: "300a1144ecb7434a9e1dd68677cab6cf" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 43095 } }
I20260923 17:27:21.561903 26961 heartbeater.cc:503] Master 127.11.229.62:41427 was elected leader, sending a full tablet report...
I20260923 17:27:21.562232 26974 ts_tablet_manager.cc:1428] T ea02f341fd68421d8a5b0e7b54b86bf5 P de8c684e115544cfa24e6fd295f49c9c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 17:27:21.562439 26975 raft_consensus.cc:348] T ea02f341fd68421d8a5b0e7b54b86bf5 P 73d1d9b41a914836a80114437655ec6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de8c684e115544cfa24e6fd295f49c9c" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35255 } } peers { permanent_uuid: "73d1d9b41a914836a80114437655ec6e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42075 } } peers { permanent_uuid: "300a1144ecb7434a9e1dd68677cab6cf" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 43095 } }
I20260923 17:27:21.562656 26975 raft_consensus.cc:374] T ea02f341fd68421d8a5b0e7b54b86bf5 P 73d1d9b41a914836a80114437655ec6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:27:21.562690 26975 raft_consensus.cc:729] T ea02f341fd68421d8a5b0e7b54b86bf5 P 73d1d9b41a914836a80114437655ec6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73d1d9b41a914836a80114437655ec6e, State: Initialized, Role: FOLLOWER
I20260923 17:27:21.563074 26975 consensus_queue.cc:260] T ea02f341fd68421d8a5b0e7b54b86bf5 P 73d1d9b41a914836a80114437655ec6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de8c684e115544cfa24e6fd295f49c9c" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35255 } } peers { permanent_uuid: "73d1d9b41a914836a80114437655ec6e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42075 } } peers { permanent_uuid: "300a1144ecb7434a9e1dd68677cab6cf" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 43095 } }
I20260923 17:27:21.564105 26975 ts_tablet_manager.cc:1428] T ea02f341fd68421d8a5b0e7b54b86bf5 P 73d1d9b41a914836a80114437655ec6e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260923 17:27:21.564354 26829 heartbeater.cc:503] Master 127.11.229.62:41427 was elected leader, sending a full tablet report...
W20260923 17:27:21.612609 26692 tablet.cc:2369] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:27:21.683960 26979 raft_consensus.cc:482] T ea02f341fd68421d8a5b0e7b54b86bf5 P 73d1d9b41a914836a80114437655ec6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:27:21.684206 26979 raft_consensus.cc:504] T ea02f341fd68421d8a5b0e7b54b86bf5 P 73d1d9b41a914836a80114437655ec6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de8c684e115544cfa24e6fd295f49c9c" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35255 } } peers { permanent_uuid: "73d1d9b41a914836a80114437655ec6e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42075 } } peers { permanent_uuid: "300a1144ecb7434a9e1dd68677cab6cf" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 43095 } }
I20260923 17:27:21.685508 26979 leader_election.cc:288] T ea02f341fd68421d8a5b0e7b54b86bf5 P 73d1d9b41a914836a80114437655ec6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de8c684e115544cfa24e6fd295f49c9c (127.11.229.3:35255), 300a1144ecb7434a9e1dd68677cab6cf (127.11.229.1:43095)
I20260923 17:27:21.689607 26970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:21.685804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:43061 (local address 127.11.229.3:35255)
0923 17:27:21.685947 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:21.685951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:21.686247 (+ 296us) server_negotiation.cc:407] Connection header received
0923 17:27:21.686430 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:21.686433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:21.686482 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:21.686564 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:27:21.686889 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.687438 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.688573 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.688980 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.689109 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:21.689164 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:21.689381 (+ 217us) server_negotiation.cc:299] Negotiation successful
0923 17:27:21.689455 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 17:27:21.689643 26980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:21.685693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:35255 (local address 127.11.229.2:43061)
0923 17:27:21.686132 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:21.686144 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:21.686317 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:21.686576 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:21.686579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:21.686595 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:27:21.686774 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:21.686780 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.687548 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.687554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:21.688445 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:21.688454 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.689105 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.689109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:21.689207 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:21.689231 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 17:27:21.689356 (+ 125us) client_negotiation.cc:240] Negotiation successful
0923 17:27:21.689456 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":74,"spinlock_wait_cycles":56192}
I20260923 17:27:21.689646 26831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:21.685779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:43095 (local address 127.11.229.2:59433)
0923 17:27:21.686163 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:21.686177 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:21.686310 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:21.686613 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:21.686616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:21.686638 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:27:21.686792 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:21.686798 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.687549 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.687554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:21.688447 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:21.688456 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.689162 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.689166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:21.689297 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:21.689319 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 17:27:21.689384 (+ 65us) client_negotiation.cc:240] Negotiation successful
0923 17:27:21.689495 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":113,"spinlock_wait_cycles":15360,"thread_start_us":125,"threads_started":1}
I20260923 17:27:21.689647 26981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:21.685896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:59433 (local address 127.11.229.1:43095)
0923 17:27:21.686167 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:21.686174 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:21.686225 (+ 51us) server_negotiation.cc:407] Connection header received
0923 17:27:21.686428 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:21.686431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:21.686494 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:21.686604 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:27:21.686908 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.687437 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.688573 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.689024 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.689177 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:21.689382 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:21.689420 (+ 38us) server_negotiation.cc:299] Negotiation successful
0923 17:27:21.689493 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":131,"threads_started":1}
I20260923 17:27:21.690469 26915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea02f341fd68421d8a5b0e7b54b86bf5" candidate_uuid: "73d1d9b41a914836a80114437655ec6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de8c684e115544cfa24e6fd295f49c9c" is_pre_election: true
I20260923 17:27:21.690600 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea02f341fd68421d8a5b0e7b54b86bf5" candidate_uuid: "73d1d9b41a914836a80114437655ec6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "300a1144ecb7434a9e1dd68677cab6cf" is_pre_election: true
I20260923 17:27:21.690819 26915 raft_consensus.cc:2454] T ea02f341fd68421d8a5b0e7b54b86bf5 P de8c684e115544cfa24e6fd295f49c9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73d1d9b41a914836a80114437655ec6e in term 0.
I20260923 17:27:21.690910 26645 raft_consensus.cc:2454] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73d1d9b41a914836a80114437655ec6e in term 0.
I20260923 17:27:21.691378 26716 leader_election.cc:302] T ea02f341fd68421d8a5b0e7b54b86bf5 P 73d1d9b41a914836a80114437655ec6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 300a1144ecb7434a9e1dd68677cab6cf, 73d1d9b41a914836a80114437655ec6e; no voters:
I20260923 17:27:21.691650 26979 raft_consensus.cc:2781] T ea02f341fd68421d8a5b0e7b54b86bf5 P 73d1d9b41a914836a80114437655ec6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:27:21.691733 26979 raft_consensus.cc:482] T ea02f341fd68421d8a5b0e7b54b86bf5 P 73d1d9b41a914836a80114437655ec6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:27:21.691785 26979 raft_consensus.cc:3037] T ea02f341fd68421d8a5b0e7b54b86bf5 P 73d1d9b41a914836a80114437655ec6e [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:27:21.692848 26979 raft_consensus.cc:504] T ea02f341fd68421d8a5b0e7b54b86bf5 P 73d1d9b41a914836a80114437655ec6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de8c684e115544cfa24e6fd295f49c9c" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35255 } } peers { permanent_uuid: "73d1d9b41a914836a80114437655ec6e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42075 } } peers { permanent_uuid: "300a1144ecb7434a9e1dd68677cab6cf" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 43095 } }
I20260923 17:27:21.693207 26979 leader_election.cc:288] T ea02f341fd68421d8a5b0e7b54b86bf5 P 73d1d9b41a914836a80114437655ec6e [CANDIDATE]: Term 1 election: Requested vote from peers de8c684e115544cfa24e6fd295f49c9c (127.11.229.3:35255), 300a1144ecb7434a9e1dd68677cab6cf (127.11.229.1:43095)
I20260923 17:27:21.693553 26915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea02f341fd68421d8a5b0e7b54b86bf5" candidate_uuid: "73d1d9b41a914836a80114437655ec6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de8c684e115544cfa24e6fd295f49c9c"
I20260923 17:27:21.693610 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea02f341fd68421d8a5b0e7b54b86bf5" candidate_uuid: "73d1d9b41a914836a80114437655ec6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "300a1144ecb7434a9e1dd68677cab6cf"
I20260923 17:27:21.693673 26915 raft_consensus.cc:3037] T ea02f341fd68421d8a5b0e7b54b86bf5 P de8c684e115544cfa24e6fd295f49c9c [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:27:21.693765 26645 raft_consensus.cc:3037] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:27:21.694701 26915 raft_consensus.cc:2454] T ea02f341fd68421d8a5b0e7b54b86bf5 P de8c684e115544cfa24e6fd295f49c9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73d1d9b41a914836a80114437655ec6e in term 1.
I20260923 17:27:21.694794 26645 raft_consensus.cc:2454] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73d1d9b41a914836a80114437655ec6e in term 1.
I20260923 17:27:21.694981 26718 leader_election.cc:302] T ea02f341fd68421d8a5b0e7b54b86bf5 P 73d1d9b41a914836a80114437655ec6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73d1d9b41a914836a80114437655ec6e, de8c684e115544cfa24e6fd295f49c9c; no voters:
I20260923 17:27:21.695143 26979 raft_consensus.cc:2781] T ea02f341fd68421d8a5b0e7b54b86bf5 P 73d1d9b41a914836a80114437655ec6e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:27:21.695436 26979 raft_consensus.cc:686] T ea02f341fd68421d8a5b0e7b54b86bf5 P 73d1d9b41a914836a80114437655ec6e [term 1 LEADER]: Becoming Leader. State: Replica: 73d1d9b41a914836a80114437655ec6e, State: Running, Role: LEADER
I20260923 17:27:21.695822 26979 consensus_queue.cc:237] T ea02f341fd68421d8a5b0e7b54b86bf5 P 73d1d9b41a914836a80114437655ec6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de8c684e115544cfa24e6fd295f49c9c" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35255 } } peers { permanent_uuid: "73d1d9b41a914836a80114437655ec6e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42075 } } peers { permanent_uuid: "300a1144ecb7434a9e1dd68677cab6cf" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 43095 } }
I20260923 17:27:21.699200 26503 catalog_manager.cc:5495] T ea02f341fd68421d8a5b0e7b54b86bf5 P 73d1d9b41a914836a80114437655ec6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 73d1d9b41a914836a80114437655ec6e (127.11.229.2). New cstate: current_term: 1 leader_uuid: "73d1d9b41a914836a80114437655ec6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de8c684e115544cfa24e6fd295f49c9c" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73d1d9b41a914836a80114437655ec6e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42075 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "300a1144ecb7434a9e1dd68677cab6cf" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 43095 } health_report { overall_health: UNKNOWN } } }
I20260923 17:27:21.728919 26972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:21.725652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46962 (local address 127.11.229.2:42075)
0923 17:27:21.725807 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:21.725810 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:21.725824 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:27:21.725907 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:21.725909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:21.725965 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:21.726067 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:27:21.726323 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.726835 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.727630 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:21.727839 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:21.727915 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:21.728183 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:27:21.728600 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:27:21.728691 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:21.728756 (+ 65us) server_negotiation.cc:299] Negotiation successful
0923 17:27:21.728809 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 17:27:21.734944 26645 raft_consensus.cc:1261] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [term 1 FOLLOWER]: Refusing update from remote peer 73d1d9b41a914836a80114437655ec6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:27:21.734951 26915 raft_consensus.cc:1261] T ea02f341fd68421d8a5b0e7b54b86bf5 P de8c684e115544cfa24e6fd295f49c9c [term 1 FOLLOWER]: Refusing update from remote peer 73d1d9b41a914836a80114437655ec6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:27:21.735745 26982 consensus_queue.cc:1035] T ea02f341fd68421d8a5b0e7b54b86bf5 P 73d1d9b41a914836a80114437655ec6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "300a1144ecb7434a9e1dd68677cab6cf" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 43095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:27:21.735975 26979 consensus_queue.cc:1035] T ea02f341fd68421d8a5b0e7b54b86bf5 P 73d1d9b41a914836a80114437655ec6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "de8c684e115544cfa24e6fd295f49c9c" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:27:21.741775 26984 mvcc.cc:205] Tried to move back new op lower bound from 7332595473338683392 to 7332595473189138432. Current Snapshot: MvccSnapshot[applied={T|T < 7332595473338683392}]
W20260923 17:27:21.742396 26962 tablet.cc:2369] T ea02f341fd68421d8a5b0e7b54b86bf5 P de8c684e115544cfa24e6fd295f49c9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:27:21.752501 26501 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:41476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:27:21.752676 26501 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:41476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:27:21.755108 26501 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311: Sending DeleteTablet for 3 replicas of tablet ea02f341fd68421d8a5b0e7b54b86bf5
I20260923 17:27:21.755823 26763 tablet_service.cc:1514] Processing DeleteTablet for tablet ea02f341fd68421d8a5b0e7b54b86bf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:27:21 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:46958
I20260923 17:27:21.755883 26625 tablet_service.cc:1514] Processing DeleteTablet for tablet ea02f341fd68421d8a5b0e7b54b86bf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:27:21 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:39976
I20260923 17:27:21.755831 26895 tablet_service.cc:1514] Processing DeleteTablet for tablet ea02f341fd68421d8a5b0e7b54b86bf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:27:21 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:52278
I20260923 17:27:21.756301 26994 tablet_replica.cc:330] T ea02f341fd68421d8a5b0e7b54b86bf5 P 73d1d9b41a914836a80114437655ec6e: stopping tablet replica
I20260923 17:27:21.756453 26995 tablet_replica.cc:330] T ea02f341fd68421d8a5b0e7b54b86bf5 P de8c684e115544cfa24e6fd295f49c9c: stopping tablet replica
I20260923 17:27:21.756604 26994 raft_consensus.cc:2229] T ea02f341fd68421d8a5b0e7b54b86bf5 P 73d1d9b41a914836a80114437655ec6e [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:27:21.756595 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 26541
I20260923 17:27:21.757300 26995 raft_consensus.cc:2229] T ea02f341fd68421d8a5b0e7b54b86bf5 P de8c684e115544cfa24e6fd295f49c9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:27:21.757628 26995 raft_consensus.cc:2258] T ea02f341fd68421d8a5b0e7b54b86bf5 P de8c684e115544cfa24e6fd295f49c9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:27:21.759157 26994 raft_consensus.cc:2258] T ea02f341fd68421d8a5b0e7b54b86bf5 P 73d1d9b41a914836a80114437655ec6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:27:21.760285 26995 ts_tablet_manager.cc:1905] T ea02f341fd68421d8a5b0e7b54b86bf5 P de8c684e115544cfa24e6fd295f49c9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:27:21.760504 26994 ts_tablet_manager.cc:1905] T ea02f341fd68421d8a5b0e7b54b86bf5 P 73d1d9b41a914836a80114437655ec6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:27:21.763119 26995 ts_tablet_manager.cc:1918] T ea02f341fd68421d8a5b0e7b54b86bf5 P de8c684e115544cfa24e6fd295f49c9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:27:21.763242 26995 log.cc:1196] T ea02f341fd68421d8a5b0e7b54b86bf5 P de8c684e115544cfa24e6fd295f49c9c: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/wal/wals/ea02f341fd68421d8a5b0e7b54b86bf5
I20260923 17:27:21.763407 26994 ts_tablet_manager.cc:1918] T ea02f341fd68421d8a5b0e7b54b86bf5 P 73d1d9b41a914836a80114437655ec6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:27:21.763564 26994 log.cc:1196] T ea02f341fd68421d8a5b0e7b54b86bf5 P 73d1d9b41a914836a80114437655ec6e: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/wal/wals/ea02f341fd68421d8a5b0e7b54b86bf5
I20260923 17:27:21.763618 26995 ts_tablet_manager.cc:1939] T ea02f341fd68421d8a5b0e7b54b86bf5 P de8c684e115544cfa24e6fd295f49c9c: Deleting consensus metadata
I20260923 17:27:21.763988 26994 ts_tablet_manager.cc:1939] T ea02f341fd68421d8a5b0e7b54b86bf5 P 73d1d9b41a914836a80114437655ec6e: Deleting consensus metadata
W20260923 17:27:21.764504 26487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.229.1:43095 (error 108)
I20260923 17:27:21.765129 26490 catalog_manager.cc:4841] TS 73d1d9b41a914836a80114437655ec6e (127.11.229.2:42075): tablet ea02f341fd68421d8a5b0e7b54b86bf5 (table test-table [id=f3d81d167124402298a8d0e83e84b789]) successfully deleted
I20260923 17:27:21.765159 26489 catalog_manager.cc:4841] TS de8c684e115544cfa24e6fd295f49c9c (127.11.229.3:35255): tablet ea02f341fd68421d8a5b0e7b54b86bf5 (table test-table [id=f3d81d167124402298a8d0e83e84b789]) successfully deleted
W20260923 17:27:21.765996 26971 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:27:21.765377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:43095 (local address 127.0.0.1:39988)
0923 17:27:21.765539 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:21.765555 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:21.765642 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:21.765915 (+ 273us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.1:43095: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":54}
I20260923 17:27:21.766194 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 26693
W20260923 17:27:21.766203 26487 catalog_manager.cc:4584] TS 300a1144ecb7434a9e1dd68677cab6cf (127.11.229.1:43095): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ea02f341fd68421d8a5b0e7b54b86bf5: Network error: Client connection negotiation failed: client connection to 127.11.229.1:43095: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 17:27:21.772331 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 26832
I20260923 17:27:21.777984 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 26470
I20260923 17:27:21.784545 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:41427
--webserver_interface=127.11.229.62
--webserver_port=43819
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:44585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:41427
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:27:21.894726 26997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:21.895016 26997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:21.895097 26997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:21.899065 26997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:27:21.899163 26997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:21.899201 26997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:21.899240 26997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:27:21.899281 26997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:27:21.904433 26997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:44585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:41427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:41427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=43819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:27:21.905546 26997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:21.906697 26997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:21.916422 27005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:21.916491 27002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:21.916635 26997 server_base.cc:1034] running on GCE node
W20260923 17:27:21.916512 27003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:21.917466 26997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:21.918555 26997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:21.919719 26997 hybrid_clock.cc:648] HybridClock initialized: now 1790184441919688 us; error 45 us; skew 500 ppm
Sep 23 17:27:21 dist-test-slave-ptl9 krb5kdc[25922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184441, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:21.922420 26997 init.cc:377] Logged in from keytab as kudu/127.11.229.62@KRBTEST.COM (short username kudu)
I20260923 17:27:21.923321 26997 webserver.cc:460] Webserver started at http://127.11.229.62:43819/ using document root <none> and password file <none>
I20260923 17:27:21.923899 26997 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:21.923981 26997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:21.927762 26997 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:27:21.929909 27012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:21.931056 26997 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:27:21.931180 26997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "c863213e6d3941c2890d8e505df89311"
format_stamp: "Formatted at 2026-09-23 17:27:19 on dist-test-slave-ptl9"
server_key: "f145fb9a8be9e7b499670d19cf3e76c7"
server_key_iv: "2e902bb5d573604c3c2ee4cff81d027e"
server_key_version: "encryptionkey@0"
I20260923 17:27:21.931715 26997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:21.952050 26997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:21.955031 26997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:21.955222 26997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:21.962461 26997 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:41427
I20260923 17:27:21.962733 27064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:41427 every 8 connection(s)
I20260923 17:27:21.963557 26997 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:27:21.967772 27065 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:27:21.969444 27065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311: Bootstrap starting.
I20260923 17:27:21.970532 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 26997
I20260923 17:27:21.971058 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.1:43095
--local_ip_for_outbound_sockets=127.11.229.1
--tserver_master_addrs=127.11.229.62:41427
--webserver_port=34821
--webserver_interface=127.11.229.1
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:44585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:27:21.988117 27065 log.cc:824] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311: Log is configured to *not* fsync() on all Append() calls
I20260923 17:27:21.999400 27065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:27:21.999783 27065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311: Bootstrap complete.
I20260923 17:27:22.004288 27065 raft_consensus.cc:348] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c863213e6d3941c2890d8e505df89311" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 41427 } }
I20260923 17:27:22.004909 27065 raft_consensus.cc:729] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c863213e6d3941c2890d8e505df89311, State: Initialized, Role: FOLLOWER
I20260923 17:27:22.005494 27065 consensus_queue.cc:260] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c863213e6d3941c2890d8e505df89311" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 41427 } }
I20260923 17:27:22.005610 27065 raft_consensus.cc:388] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:27:22.005704 27065 raft_consensus.cc:482] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:27:22.005821 27065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [term 2 FOLLOWER]: Advancing to term 3
I20260923 17:27:22.007833 27065 raft_consensus.cc:504] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c863213e6d3941c2890d8e505df89311" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 41427 } }
I20260923 17:27:22.008162 27065 leader_election.cc:302] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c863213e6d3941c2890d8e505df89311; no voters:
I20260923 17:27:22.008486 27065 leader_election.cc:288] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 17:27:22.008566 27069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [term 3 FOLLOWER]: Leader election won for term 3
I20260923 17:27:22.008801 27069 raft_consensus.cc:686] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [term 3 LEADER]: Becoming Leader. State: Replica: c863213e6d3941c2890d8e505df89311, State: Running, Role: LEADER
I20260923 17:27:22.009091 27069 consensus_queue.cc:237] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c863213e6d3941c2890d8e505df89311" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 41427 } }
I20260923 17:27:22.009940 27065 sys_catalog.cc:564] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:27:22.010056 27069 sys_catalog.cc:455] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c863213e6d3941c2890d8e505df89311" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c863213e6d3941c2890d8e505df89311" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 41427 } } }
I20260923 17:27:22.010152 27069 sys_catalog.cc:458] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:22.010453 27070 sys_catalog.cc:455] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c863213e6d3941c2890d8e505df89311. Latest consensus state: current_term: 3 leader_uuid: "c863213e6d3941c2890d8e505df89311" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c863213e6d3941c2890d8e505df89311" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 41427 } } }
I20260923 17:27:22.010574 27070 sys_catalog.cc:458] T 00000000000000000000000000000000 P c863213e6d3941c2890d8e505df89311 [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:22.010509 27073 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:27:22.016845 27073 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:27:22.017699 27073 catalog_manager.cc:1259] Loaded cluster ID: 66a085f4ceb9403bbb8e345b2a1ad23f
I20260923 17:27:22.017786 27073 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260923 17:27:22.018612 27073 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe4aa7359ad google::LogMessage::Fail() at ??:0
@ 0x7fe4aa73888a google::LogMessage::SendToLog() at ??:0
@ 0x7fe4aa7354de google::LogMessage::Flush() at ??:0
@ 0x7fe4aa736a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe4b5074fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fe4b5075d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe4b5072579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fe4b50b6640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe4ac9873d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe4ac987c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fe4ac9893f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe4ac978512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe4adfbc6db start_thread at ??:0
@ 0x7fe4a923361f clone at ??:0
W20260923 17:27:22.116143 27067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:22.116726 27067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:22.116868 27067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:22.123800 27067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:22.123893 27067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:22.123986 27067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.1
I20260923 17:27:22.132393 27067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:44585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.1:43095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.229.1
--webserver_port=34821
--tserver_master_addrs=127.11.229.62:41427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.1
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:27:22.133955 27067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:22.135243 27067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:22.146683 27109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:22.146692 27106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:22.146692 27107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:22.147045 27067 server_base.cc:1034] running on GCE node
I20260923 17:27:22.147524 27067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:22.148172 27067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:22.149353 27067 hybrid_clock.cc:648] HybridClock initialized: now 1790184442149326 us; error 46 us; skew 500 ppm
Sep 23 17:27:22 dist-test-slave-ptl9 krb5kdc[25922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184442, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:22.152182 27067 init.cc:377] Logged in from keytab as kudu/127.11.229.1@KRBTEST.COM (short username kudu)
I20260923 17:27:22.153072 27067 webserver.cc:460] Webserver started at http://127.11.229.1:34821/ using document root <none> and password file <none>
I20260923 17:27:22.153694 27067 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:22.153756 27067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:22.157725 27067 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:27:22.159919 27116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:22.161439 27067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:27:22.161571 27067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/wal
uuid: "300a1144ecb7434a9e1dd68677cab6cf"
format_stamp: "Formatted at 2026-09-23 17:27:19 on dist-test-slave-ptl9"
server_key: "3cae6515633ab0f53fe1de8dace8a170"
server_key_iv: "975ef45b8786364365e90e24a6b7f52c"
server_key_version: "encryptionkey@0"
I20260923 17:27:22.162171 27067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:22.171531 27067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:22.174651 27067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:22.174852 27067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:22.175431 27067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:27:22.176543 27123 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 17:27:22.181644 27067 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 17:27:22.181720 27067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260923 17:27:22.181802 27067 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 17:27:22.183455 27067 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 17:27:22.183523 27067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260923 17:27:22.183781 27123 tablet_bootstrap.cc:492] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf: Bootstrap starting.
I20260923 17:27:22.194455 27067 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.1:43095
I20260923 17:27:22.194497 27230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.1:43095 every 8 connection(s)
I20260923 17:27:22.195604 27067 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
I20260923 17:27:22.198141 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 27067
I20260923 17:27:22.198616 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.2:42075
--local_ip_for_outbound_sockets=127.11.229.2
--tserver_master_addrs=127.11.229.62:41427
--webserver_port=35055
--webserver_interface=127.11.229.2
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:44585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:27:22.204154 27123 log.cc:824] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf: Log is configured to *not* fsync() on all Append() calls
I20260923 17:27:22.211855 27123 tablet_bootstrap.cc:492] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:27:22.212226 27123 tablet_bootstrap.cc:492] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf: Bootstrap complete.
I20260923 17:27:22.213455 27123 ts_tablet_manager.cc:1397] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf: Time spent bootstrapping tablet: real 0.030s user 0.026s sys 0.003s
I20260923 17:27:22.216218 27123 raft_consensus.cc:348] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de8c684e115544cfa24e6fd295f49c9c" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35255 } } peers { permanent_uuid: "73d1d9b41a914836a80114437655ec6e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42075 } } peers { permanent_uuid: "300a1144ecb7434a9e1dd68677cab6cf" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 43095 } }
I20260923 17:27:22.216789 27123 raft_consensus.cc:729] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 300a1144ecb7434a9e1dd68677cab6cf, State: Initialized, Role: FOLLOWER
I20260923 17:27:22.217386 27123 consensus_queue.cc:260] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de8c684e115544cfa24e6fd295f49c9c" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35255 } } peers { permanent_uuid: "73d1d9b41a914836a80114437655ec6e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42075 } } peers { permanent_uuid: "300a1144ecb7434a9e1dd68677cab6cf" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 43095 } }
I20260923 17:27:22.218361 27123 ts_tablet_manager.cc:1428] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260923 17:27:22.352437 27234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:22.352797 27234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:22.352869 27234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:22.358868 27234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:22.358950 27234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:22.359035 27234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.2
I20260923 17:27:22.367174 27234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:44585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.2:42075
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.229.2
--webserver_port=35055
--tserver_master_addrs=127.11.229.62:41427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.2
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:27:22.368687 27234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:22.370093 27234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:22.383392 27241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:22.383392 27242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:22.383677 27234 server_base.cc:1034] running on GCE node
W20260923 17:27:22.383913 27244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:22.384320 27234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:22.384990 27234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:22.386190 27234 hybrid_clock.cc:648] HybridClock initialized: now 1790184442386174 us; error 40 us; skew 500 ppm
W20260923 17:27:22.386516 27233 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:27:22.198055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.1:37017)
0923 17:27:22.198598 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:22.198652 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:22.199673 (+ 1021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:22.385591 (+185918us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":176,"threads_started":1}
W20260923 17:27:22.387046 27231 heartbeater.cc:644] Failed to heartbeat to 127.11.229.62:41427 (0 consecutive failures): Network error: Failed to ping master at 127.11.229.62:41427: Client connection negotiation failed: client connection to 127.11.229.62:41427: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 17:27:22.387780 27233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:22.387522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.1:60237)
0923 17:27:22.387658 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:22.387722 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260923 17:27:22.388437 27233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:22.388223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.1:60351)
0923 17:27:22.388336 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:22.388380 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 17:27:22.388630 27231 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 23 17:27:22 dist-test-slave-ptl9 krb5kdc[25922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184442, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:22.389365 27234 init.cc:377] Logged in from keytab as kudu/127.11.229.2@KRBTEST.COM (short username kudu)
I20260923 17:27:22.390178 27234 webserver.cc:460] Webserver started at http://127.11.229.2:35055/ using document root <none> and password file <none>
I20260923 17:27:22.390703 27234 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:22.390775 27234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:22.394564 27234 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:27:22.396515 27251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:22.397877 27234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:27:22.397970 27234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/wal
uuid: "73d1d9b41a914836a80114437655ec6e"
format_stamp: "Formatted at 2026-09-23 17:27:20 on dist-test-slave-ptl9"
server_key: "4431f6672aa0ddca05bc42c366860378"
server_key_iv: "beae22808722dc86821f544c122d1c4b"
server_key_version: "encryptionkey@0"
I20260923 17:27:22.398485 27234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:22.410004 27234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:22.413251 27234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:22.413488 27234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:22.414019 27234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:27:22.414788 27234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:27:22.414831 27234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:22.414870 27234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:27:22.414886 27234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:22.423913 27234 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.2:42075
I20260923 17:27:22.423933 27364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.2:42075 every 8 connection(s)
I20260923 17:27:22.425009 27234 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
I20260923 17:27:22.428258 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 27234
I20260923 17:27:22.428659 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.3:35255
--local_ip_for_outbound_sockets=127.11.229.3
--tserver_master_addrs=127.11.229.62:41427
--webserver_port=42641
--webserver_interface=127.11.229.3
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:44585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:27:22.429175 27367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:22.427267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.2:44049)
0923 17:27:22.427729 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:22.427853 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":115,"threads_started":1}
W20260923 17:27:22.429709 27365 heartbeater.cc:644] Failed to heartbeat to 127.11.229.62:41427 (0 consecutive failures): Network error: Failed to ping master at 127.11.229.62:41427: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
I20260923 17:27:22.430400 27367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:22.430167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.2:40885)
0923 17:27:22.430294 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:22.430347 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260923 17:27:22.431030 27367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:22.430802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.2:57279)
0923 17:27:22.430926 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:22.430971 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260923 17:27:22.431193 27365 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 17:27:22.544225 27368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:22.544468 27368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:22.544513 27368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:22.548588 27368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:22.548650 27368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:22.548743 27368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.3
I20260923 17:27:22.554111 27368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:44585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.3:35255
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.229.3
--webserver_port=42641
--tserver_master_addrs=127.11.229.62:41427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.3
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:27:22.555200 27368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:22.556226 27368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:22.565876 27373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:22.565883 27374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:22.565893 27376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:22.566080 27368 server_base.cc:1034] running on GCE node
I20260923 17:27:22.566509 27368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:22.567167 27368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:22.568336 27368 hybrid_clock.cc:648] HybridClock initialized: now 1790184442568320 us; error 25 us; skew 500 ppm
Sep 23 17:27:22 dist-test-slave-ptl9 krb5kdc[25922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184442, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:22.571168 27368 init.cc:377] Logged in from keytab as kudu/127.11.229.3@KRBTEST.COM (short username kudu)
I20260923 17:27:22.572014 27368 webserver.cc:460] Webserver started at http://127.11.229.3:42641/ using document root <none> and password file <none>
I20260923 17:27:22.572595 27368 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:22.572656 27368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:22.576637 27368 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:27:22.578764 27383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:22.579980 27368 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:27:22.580099 27368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/wal
uuid: "de8c684e115544cfa24e6fd295f49c9c"
format_stamp: "Formatted at 2026-09-23 17:27:20 on dist-test-slave-ptl9"
server_key: "3baced430a71467226be102e91ac1056"
server_key_iv: "99e00b927367ff9332dedf682ce6f80e"
server_key_version: "encryptionkey@0"
I20260923 17:27:22.580641 27368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:22.599001 27368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:22.602200 27368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:22.602421 27368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:22.603121 27368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:27:22.603962 27368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:27:22.604008 27368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:22.604077 27368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:27:22.604117 27368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:22.613072 27368 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.3:35255
I20260923 17:27:22.613215 27496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.3:35255 every 8 connection(s)
I20260923 17:27:22.614151 27368 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
I20260923 17:27:22.615501 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 27368
I20260923 17:27:22.617740 27500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:22.616406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.3:52445)
0923 17:27:22.616871 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:22.616974 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":146,"threads_started":1}
W20260923 17:27:22.618196 27497 heartbeater.cc:644] Failed to heartbeat to 127.11.229.62:41427 (0 consecutive failures): Network error: Failed to ping master at 127.11.229.62:41427: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
I20260923 17:27:22.618736 27500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:22.618541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.3:50143)
0923 17:27:22.618638 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:22.618682 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260923 17:27:22.619349 27500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:22.619172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.3:60679)
0923 17:27:22.619260 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:22.619307 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260923 17:27:22.619571 27497 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 17:27:23.389963 27501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:23.389441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.1:58041)
0923 17:27:23.389708 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:23.389790 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":92,"threads_started":1}
I20260923 17:27:23.432333 27502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:23.431858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.2:51429)
0923 17:27:23.432162 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:23.432227 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":72,"threads_started":1}
I20260923 17:27:23.620563 27503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:23.620135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.3:39387)
0923 17:27:23.620411 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:23.620462 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":105,"threads_started":1}
I20260923 17:27:24.072670 27504 raft_consensus.cc:482] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:27:24.072904 27504 raft_consensus.cc:504] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de8c684e115544cfa24e6fd295f49c9c" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35255 } } peers { permanent_uuid: "73d1d9b41a914836a80114437655ec6e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42075 } } peers { permanent_uuid: "300a1144ecb7434a9e1dd68677cab6cf" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 43095 } }
I20260923 17:27:24.074121 27504 leader_election.cc:288] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de8c684e115544cfa24e6fd295f49c9c (127.11.229.3:35255), 73d1d9b41a914836a80114437655ec6e (127.11.229.2:42075)
Sep 23 17:27:24 dist-test-slave-ptl9 krb5kdc[25922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184442, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for kudu/127.11.229.3@KRBTEST.COM
Sep 23 17:27:24 dist-test-slave-ptl9 krb5kdc[25922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184442, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for kudu/127.11.229.2@KRBTEST.COM
I20260923 17:27:24.088255 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:24.074358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:35255 (local address 127.11.229.1:35903)
0923 17:27:24.074680 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:24.074695 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:24.074824 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:24.076414 (+ 1590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:24.076419 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:24.076960 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:24.077446 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:24.077461 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:24.079378 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:24.079381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:24.080343 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:24.080352 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:24.081090 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:24.081093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:24.081341 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:24.081993 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:24.082009 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:24.083551 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:24.085922 (+ 2371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:24.085928 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:24.085948 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:24.086182 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:24.086536 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:24.086540 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:24.086542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:24.086671 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:24.087298 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:24.087305 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:24.087518 (+ 213us) client_negotiation.cc:769] Sending connection context
0923 17:27:24.087717 (+ 199us) client_negotiation.cc:240] Negotiation successful
0923 17:27:24.088057 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":175,"spinlock_wait_cycles":26368,"thread_start_us":75,"threads_started":1}
I20260923 17:27:24.088301 27507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:24.074553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:53321 (local address 127.11.229.2:42075)
0923 17:27:24.074985 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:24.074998 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:24.075034 (+ 36us) server_negotiation.cc:407] Connection header received
0923 17:27:24.075701 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:24.075707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:24.076052 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:24.076362 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:24.077644 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:24.079227 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:24.080484 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:24.080963 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:24.081147 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:24.084099 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:24.084106 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:24.084123 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:24.084152 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:24.085997 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:24.086442 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:24.086447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:24.086453 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:24.086567 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:24.086854 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:24.086859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:24.086861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:24.087235 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:24.087374 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:24.087819 (+ 445us) server_negotiation.cc:299] Negotiation successful
0923 17:27:24.088104 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":79,"threads_started":1}
I20260923 17:27:24.088276 27508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:24.074543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:35903 (local address 127.11.229.3:35255)
0923 17:27:24.075029 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:24.075048 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:24.075078 (+ 30us) server_negotiation.cc:407] Connection header received
0923 17:27:24.075731 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:24.075736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:24.076052 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:24.076362 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:24.077642 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:24.079227 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:24.080472 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:24.080963 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:24.081146 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:24.083840 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:24.083845 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:24.083859 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:24.083886 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:24.085799 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:24.086307 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:24.086313 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:24.086318 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:24.086439 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:24.086765 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:24.086770 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:24.086771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:24.087171 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:24.087319 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:24.087824 (+ 505us) server_negotiation.cc:299] Negotiation successful
0923 17:27:24.088075 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":160,"threads_started":1}
I20260923 17:27:24.088294 27505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:24.074377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:42075 (local address 127.11.229.1:53321)
0923 17:27:24.074778 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:24.074792 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:24.074891 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:24.076414 (+ 1523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:24.076419 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:24.076959 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:24.077447 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:24.077461 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:24.079413 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:24.079416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:24.080346 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:24.080356 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:24.081090 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:24.081093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:24.081345 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:24.082050 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:24.082062 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:24.083876 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:24.086126 (+ 2250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:24.086129 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:24.086131 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:24.086328 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:24.086663 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:24.086667 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:24.086669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:24.086762 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:24.087342 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:24.087346 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:24.087520 (+ 174us) client_negotiation.cc:769] Sending connection context
0923 17:27:24.087717 (+ 197us) client_negotiation.cc:240] Negotiation successful
0923 17:27:24.088141 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":239,"spinlock_wait_cycles":3456,"thread_start_us":61,"threads_started":1}
W20260923 17:27:24.091495 27119 leader_election.cc:341] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de8c684e115544cfa24e6fd295f49c9c (127.11.229.3:35255): Not found: Tablet not found: ea02f341fd68421d8a5b0e7b54b86bf5
W20260923 17:27:24.092002 27120 leader_election.cc:341] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 73d1d9b41a914836a80114437655ec6e (127.11.229.2:42075): Not found: Tablet not found: ea02f341fd68421d8a5b0e7b54b86bf5
I20260923 17:27:24.092096 27120 leader_election.cc:302] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 300a1144ecb7434a9e1dd68677cab6cf; no voters: 73d1d9b41a914836a80114437655ec6e, de8c684e115544cfa24e6fd295f49c9c
I20260923 17:27:24.092393 27504 raft_consensus.cc:2726] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 17:27:24.391103 27505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:24.390756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.1:56383)
0923 17:27:24.390944 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:24.391006 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260923 17:27:24.433835 27509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:24.433324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.2:49057)
0923 17:27:24.433635 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:24.433709 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":89,"threads_started":1}
I20260923 17:27:24.621934 27510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:24.621471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.3:36995)
0923 17:27:24.621759 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:24.621829 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":91,"threads_started":1}
I20260923 17:27:25.392489 27511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:25.391994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.1:51111)
0923 17:27:25.392299 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:25.392375 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":84,"threads_started":1}
I20260923 17:27:25.435173 27512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:25.434721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.2:47695)
0923 17:27:25.435010 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:25.435075 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":75,"threads_started":1}
I20260923 17:27:25.623330 27513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:25.622847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.3:43855)
0923 17:27:25.623166 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:25.623234 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":80,"threads_started":1}
I20260923 17:27:25.653543 27514 raft_consensus.cc:482] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:27:25.653692 27514 raft_consensus.cc:504] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de8c684e115544cfa24e6fd295f49c9c" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35255 } } peers { permanent_uuid: "73d1d9b41a914836a80114437655ec6e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42075 } } peers { permanent_uuid: "300a1144ecb7434a9e1dd68677cab6cf" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 43095 } }
I20260923 17:27:25.654265 27514 leader_election.cc:288] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de8c684e115544cfa24e6fd295f49c9c (127.11.229.3:35255), 73d1d9b41a914836a80114437655ec6e (127.11.229.2:42075)
W20260923 17:27:25.655328 27120 leader_election.cc:341] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 73d1d9b41a914836a80114437655ec6e (127.11.229.2:42075): Not found: Tablet not found: ea02f341fd68421d8a5b0e7b54b86bf5
W20260923 17:27:25.655438 27119 leader_election.cc:341] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de8c684e115544cfa24e6fd295f49c9c (127.11.229.3:35255): Not found: Tablet not found: ea02f341fd68421d8a5b0e7b54b86bf5
I20260923 17:27:25.655505 27119 leader_election.cc:302] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 300a1144ecb7434a9e1dd68677cab6cf; no voters: 73d1d9b41a914836a80114437655ec6e, de8c684e115544cfa24e6fd295f49c9c
I20260923 17:27:25.655663 27514 raft_consensus.cc:2726] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 17:27:26.393950 27515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:26.393481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.1:44177)
0923 17:27:26.393782 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:26.393849 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":80,"threads_started":1}
I20260923 17:27:26.436583 27516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:26.436079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.2:52653)
0923 17:27:26.436424 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:26.436486 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":142,"threads_started":1}
I20260923 17:27:26.624650 27517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:26.624210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.3:34139)
0923 17:27:26.624471 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:26.624527 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":77,"threads_started":1}
I20260923 17:27:27.395617 27518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:27.395004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.1:47155)
0923 17:27:27.395395 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:27.395489 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":114,"threads_started":1}
I20260923 17:27:27.438036 27519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:27.437580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.2:59563)
0923 17:27:27.437878 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:27.437938 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":83,"threads_started":1}
I20260923 17:27:27.626027 27520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:27.625505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.3:60281)
0923 17:27:27.625865 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:27.625928 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":125,"threads_started":1}
I20260923 17:27:28.224448 27521 raft_consensus.cc:482] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:27:28.224589 27521 raft_consensus.cc:504] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de8c684e115544cfa24e6fd295f49c9c" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35255 } } peers { permanent_uuid: "73d1d9b41a914836a80114437655ec6e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42075 } } peers { permanent_uuid: "300a1144ecb7434a9e1dd68677cab6cf" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 43095 } }
I20260923 17:27:28.225018 27521 leader_election.cc:288] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de8c684e115544cfa24e6fd295f49c9c (127.11.229.3:35255), 73d1d9b41a914836a80114437655ec6e (127.11.229.2:42075)
W20260923 17:27:28.225857 27120 leader_election.cc:341] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 73d1d9b41a914836a80114437655ec6e (127.11.229.2:42075): Not found: Tablet not found: ea02f341fd68421d8a5b0e7b54b86bf5
W20260923 17:27:28.225971 27119 leader_election.cc:341] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de8c684e115544cfa24e6fd295f49c9c (127.11.229.3:35255): Not found: Tablet not found: ea02f341fd68421d8a5b0e7b54b86bf5
I20260923 17:27:28.226085 27119 leader_election.cc:302] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 300a1144ecb7434a9e1dd68677cab6cf; no voters: 73d1d9b41a914836a80114437655ec6e, de8c684e115544cfa24e6fd295f49c9c
I20260923 17:27:28.226267 27521 raft_consensus.cc:2726] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 17:27:28.396808 27522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:28.396368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.1:59693)
0923 17:27:28.396647 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:28.396704 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":86,"threads_started":1}
I20260923 17:27:28.439261 27523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:28.438809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.2:46539)
0923 17:27:28.439096 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:28.439163 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":90,"threads_started":1}
I20260923 17:27:28.627197 27524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:28.626780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.3:50431)
0923 17:27:28.627046 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:28.627104 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":88,"threads_started":1}
I20260923 17:27:29.398283 27525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:29.397760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.1:60819)
0923 17:27:29.398049 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:29.398153 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":74,"threads_started":1}
I20260923 17:27:29.440637 27526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:29.440168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.2:52903)
0923 17:27:29.440461 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:29.440519 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":83,"threads_started":1}
I20260923 17:27:29.628508 27527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:29.628001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.3:34915)
0923 17:27:29.628349 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:29.628410 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":120,"threads_started":1}
I20260923 17:27:30.208222 27528 raft_consensus.cc:482] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:27:30.208346 27528 raft_consensus.cc:504] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de8c684e115544cfa24e6fd295f49c9c" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35255 } } peers { permanent_uuid: "73d1d9b41a914836a80114437655ec6e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42075 } } peers { permanent_uuid: "300a1144ecb7434a9e1dd68677cab6cf" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 43095 } }
I20260923 17:27:30.208863 27528 leader_election.cc:288] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de8c684e115544cfa24e6fd295f49c9c (127.11.229.3:35255), 73d1d9b41a914836a80114437655ec6e (127.11.229.2:42075)
W20260923 17:27:30.209686 27119 leader_election.cc:341] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de8c684e115544cfa24e6fd295f49c9c (127.11.229.3:35255): Not found: Tablet not found: ea02f341fd68421d8a5b0e7b54b86bf5
W20260923 17:27:30.209831 27120 leader_election.cc:341] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 73d1d9b41a914836a80114437655ec6e (127.11.229.2:42075): Not found: Tablet not found: ea02f341fd68421d8a5b0e7b54b86bf5
I20260923 17:27:30.209940 27120 leader_election.cc:302] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 300a1144ecb7434a9e1dd68677cab6cf; no voters: 73d1d9b41a914836a80114437655ec6e, de8c684e115544cfa24e6fd295f49c9c
I20260923 17:27:30.210125 27528 raft_consensus.cc:2726] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 17:27:30.399644 27529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:30.399164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.1:33253)
0923 17:27:30.399468 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:30.399529 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":97,"threads_started":1}
I20260923 17:27:30.441907 27530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:30.441466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.2:34817)
0923 17:27:30.441739 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:30.441798 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":74,"threads_started":1}
I20260923 17:27:30.629915 27531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:30.629412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.3:44499)
0923 17:27:30.629672 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:30.629746 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":71,"threads_started":1}
I20260923 17:27:31.400933 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:31.400459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.1:49811)
0923 17:27:31.400771 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:31.400832 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":75,"threads_started":1}
I20260923 17:27:31.443275 27533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:31.442785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.2:60835)
0923 17:27:31.443100 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:31.443170 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":104,"threads_started":1}
I20260923 17:27:31.631075 27534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:31.630676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.3:44715)
0923 17:27:31.630923 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:31.630978 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":83,"threads_started":1}
I20260923 17:27:32.197912 27535 raft_consensus.cc:482] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:27:32.198047 27535 raft_consensus.cc:504] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de8c684e115544cfa24e6fd295f49c9c" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35255 } } peers { permanent_uuid: "73d1d9b41a914836a80114437655ec6e" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42075 } } peers { permanent_uuid: "300a1144ecb7434a9e1dd68677cab6cf" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 43095 } }
I20260923 17:27:32.198410 27535 leader_election.cc:288] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de8c684e115544cfa24e6fd295f49c9c (127.11.229.3:35255), 73d1d9b41a914836a80114437655ec6e (127.11.229.2:42075)
W20260923 17:27:32.199275 27119 leader_election.cc:341] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de8c684e115544cfa24e6fd295f49c9c (127.11.229.3:35255): Not found: Tablet not found: ea02f341fd68421d8a5b0e7b54b86bf5
W20260923 17:27:32.199412 27120 leader_election.cc:341] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 73d1d9b41a914836a80114437655ec6e (127.11.229.2:42075): Not found: Tablet not found: ea02f341fd68421d8a5b0e7b54b86bf5
I20260923 17:27:32.199507 27120 leader_election.cc:302] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 300a1144ecb7434a9e1dd68677cab6cf; no voters: 73d1d9b41a914836a80114437655ec6e, de8c684e115544cfa24e6fd295f49c9c
I20260923 17:27:32.199651 27535 raft_consensus.cc:2726] T ea02f341fd68421d8a5b0e7b54b86bf5 P 300a1144ecb7434a9e1dd68677cab6cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 17:27:32.402778 27536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:32.402208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.1:56489)
0923 17:27:32.402571 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:32.402641 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":115,"threads_started":1}
I20260923 17:27:32.444654 27537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:32.444167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.2:42431)
0923 17:27:32.444480 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:32.444549 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":96,"threads_started":1}
I20260923 17:27:32.632400 27538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:32.631941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.3:32921)
0923 17:27:32.632246 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:32.632303 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":93,"threads_started":1}
I20260923 17:27:33.404085 27539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:33.403609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.1:35547)
0923 17:27:33.403913 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:33.403977 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":77,"threads_started":1}
I20260923 17:27:33.445917 27540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:33.445514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.2:35935)
0923 17:27:33.445758 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:33.445819 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":72,"threads_started":1}
I20260923 17:27:33.633782 27541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:33.633252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.3:53333)
0923 17:27:33.633619 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:33.633682 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":93,"threads_started":1}
I20260923 17:27:34.405550 27542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:34.405031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.1:36103)
0923 17:27:34.405383 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:34.405446 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":74,"threads_started":1}
I20260923 17:27:34.447250 27543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:34.446793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.2:36437)
0923 17:27:34.447070 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:34.447140 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":91,"threads_started":1}
I20260923 17:27:34.635056 27544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:34.634594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.3:59221)
0923 17:27:34.634883 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:34.634943 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":81,"threads_started":1}
I20260923 17:27:35.407054 27545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:35.406541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.1:47513)
0923 17:27:35.406846 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:35.406939 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":119,"threads_started":1}
I20260923 17:27:35.448737 27546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:35.448221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.2:38959)
0923 17:27:35.448548 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:35.448616 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":115,"threads_started":1}
I20260923 17:27:35.636446 27547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:35.636000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.3:36113)
0923 17:27:35.636280 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:35.636346 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":83,"threads_started":1}
I20260923 17:27:36.408339 27548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:36.407903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.1:34845)
0923 17:27:36.408181 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:36.408239 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":89,"threads_started":1}
I20260923 17:27:36.450297 27549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:36.449729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.2:36385)
0923 17:27:36.450057 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:36.450168 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":98,"threads_started":1}
I20260923 17:27:36.637818 27550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:36.637383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.3:56763)
0923 17:27:36.637665 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:36.637719 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":93,"threads_started":1}
I20260923 17:27:37.409972 27551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:37.409509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.1:59721)
0923 17:27:37.409811 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:37.409872 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":102,"threads_started":1}
I20260923 17:27:37.452330 27552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:37.451668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:41427 (local address 127.11.229.2:45275)
0923 17:27:37.452092 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:37.452191 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:41427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":117,"threads_started":1}
I20260923 17:27:37.623080 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 27067
I20260923 17:27:37.630064 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 27234
I20260923 17:27:37.636861 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 27368
2026-09-23T17:27:37Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19243 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:27:37 dist-test-slave-ptl9 krb5kdc[27555](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:27:37 dist-test-slave-ptl9 krb5kdc[27555](info): set up 2 sockets
Sep 23 17:27:37 dist-test-slave-ptl9 krb5kdc[27555](info): commencing operation
krb5kdc: starting...
W20260923 17:27:38.705912 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:27:38 dist-test-slave-ptl9 krb5kdc[27555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:27:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:27:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.229.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
I20260923 17:27:38.824368 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:34709
--webserver_interface=127.11.229.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:41009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:34709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:27:38.935720 27568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:38.935992 27568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:38.936033 27568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:38.940066 27568 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 17:27:38.940120 27568 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 17:27:38.940141 27568 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 17:27:38.940156 27568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:27:38.940174 27568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:38.940236 27568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:38.940267 27568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:27:38.940295 27568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:27:38.945632 27568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:41009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:34709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:34709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:27:38.946715 27568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:38.947803 27568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 17:27:38.957310 27568 server_base.cc:1034] running on GCE node
W20260923 17:27:38.957353 27574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:38.957348 27576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:38.957564 27573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:38.958199 27568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:38.959457 27568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:38.960662 27568 hybrid_clock.cc:648] HybridClock initialized: now 1790184458960628 us; error 52 us; skew 500 ppm
Sep 23 17:27:38 dist-test-slave-ptl9 krb5kdc[27555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184458, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:38.963652 27568 init.cc:377] Logged in from keytab as kudu/127.11.229.62@KRBTEST.COM (short username kudu)
I20260923 17:27:38.964742 27568 webserver.cc:460] Webserver started at http://127.11.229.62:38351/ using document root <none> and password file <none>
I20260923 17:27:38.965546 27568 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:38.965643 27568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:38.965890 27568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:27:38.968325 27568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "0aa575a5a84b41b194b099dc7e71f04e"
format_stamp: "Formatted at 2026-09-23 17:27:38 on dist-test-slave-ptl9"
server_key: "59c03647ad7833031233bee6ca0e32be"
server_key_iv: "a60248ed97d608168500c8060557ef64"
server_key_version: "encryptionkey@0"
I20260923 17:27:38.968983 27568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "0aa575a5a84b41b194b099dc7e71f04e"
format_stamp: "Formatted at 2026-09-23 17:27:38 on dist-test-slave-ptl9"
server_key: "59c03647ad7833031233bee6ca0e32be"
server_key_iv: "a60248ed97d608168500c8060557ef64"
server_key_version: "encryptionkey@0"
I20260923 17:27:38.973522 27568 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:27:38.976266 27583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:38.978338 27568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:27:38.978598 27568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "0aa575a5a84b41b194b099dc7e71f04e"
format_stamp: "Formatted at 2026-09-23 17:27:38 on dist-test-slave-ptl9"
server_key: "59c03647ad7833031233bee6ca0e32be"
server_key_iv: "a60248ed97d608168500c8060557ef64"
server_key_version: "encryptionkey@0"
I20260923 17:27:38.978772 27568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:38.991984 27568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:38.995020 27568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:38.995221 27568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:39.002584 27568 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:34709
I20260923 17:27:39.002871 27635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:34709 every 8 connection(s)
I20260923 17:27:39.003662 27568 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:27:39.006639 27636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:27:39.010434 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 27568
I20260923 17:27:39.010536 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:27:39.010874 12180 external_mini_cluster.cc:1428] Setting key 73ea1c6d87521929381994cce0241894
I20260923 17:27:39.012939 27636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e: Bootstrap starting.
I20260923 17:27:39.015795 27636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e: Neither blocks nor log segments found. Creating new log.
I20260923 17:27:39.016675 27636 log.cc:824] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e: Log is configured to *not* fsync() on all Append() calls
I20260923 17:27:39.018957 27636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e: No bootstrap required, opened a new log
Sep 23 17:27:39 dist-test-slave-ptl9 krb5kdc[27555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:27:39.023262 27636 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa575a5a84b41b194b099dc7e71f04e" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34709 } }
I20260923 17:27:39.023571 27636 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:27:39.023629 27636 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aa575a5a84b41b194b099dc7e71f04e, State: Initialized, Role: FOLLOWER
I20260923 17:27:39.024143 27636 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa575a5a84b41b194b099dc7e71f04e" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34709 } }
I20260923 17:27:39.024328 27636 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:27:39.024410 27636 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:27:39.024514 27636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:27:39.025731 27636 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa575a5a84b41b194b099dc7e71f04e" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34709 } }
I20260923 17:27:39.026126 27636 leader_election.cc:302] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0aa575a5a84b41b194b099dc7e71f04e; no voters:
I20260923 17:27:39.026492 27636 leader_election.cc:288] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:27:39.026629 27641 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:27:39.026894 27641 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [term 1 LEADER]: Becoming Leader. State: Replica: 0aa575a5a84b41b194b099dc7e71f04e, State: Running, Role: LEADER
I20260923 17:27:39.027251 27641 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa575a5a84b41b194b099dc7e71f04e" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34709 } }
I20260923 17:27:39.027541 27639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:39.012329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43410 (local address 127.11.229.62:34709)
0923 17:27:39.012769 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:39.012790 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:39.012834 (+ 44us) server_negotiation.cc:407] Connection header received
0923 17:27:39.013726 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:39.013750 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:39.014181 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:39.014559 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:39.015223 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:39.016785 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:39.017866 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:39.018536 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:39.018723 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:39.021366 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:39.021375 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:39.021393 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:39.021426 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:39.024048 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:39.024529 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:39.024536 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:39.024542 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:39.024701 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:39.024974 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:39.024977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:39.024979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:39.025531 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:39.025741 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:39.026019 (+ 278us) server_negotiation.cc:299] Negotiation successful
0923 17:27:39.026332 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":113,"threads_started":1}
I20260923 17:27:39.028695 27642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0aa575a5a84b41b194b099dc7e71f04e. Latest consensus state: current_term: 1 leader_uuid: "0aa575a5a84b41b194b099dc7e71f04e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa575a5a84b41b194b099dc7e71f04e" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34709 } } }
I20260923 17:27:39.028911 27636 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:27:39.028918 27642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:39.029011 27643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0aa575a5a84b41b194b099dc7e71f04e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa575a5a84b41b194b099dc7e71f04e" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34709 } } }
I20260923 17:27:39.029153 27643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [sys.catalog]: This master's current role is: LEADER
I20260923 17:27:39.030373 27645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:27:39.033212 27645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:27:39.038352 27645 catalog_manager.cc:1347] Generated new cluster ID: e7a40a4a7b144207bdc56630af64e984
I20260923 17:27:39.038419 27645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:27:39.051961 27645 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:27:39.052959 27645 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:27:39.059791 27645 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e: Generated new TSK 0
I20260923 17:27:39.060823 27645 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.229.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab.
Entry for principal kudu/127.11.229.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab.
I20260923 17:27:39.092023 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.1:0
--local_ip_for_outbound_sockets=127.11.229.1
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:34709
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:41009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:27:39.205673 27662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:39.205981 27662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:39.206059 27662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:39.210150 27662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:39.210237 27662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:39.210331 27662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.1
I20260923 17:27:39.215759 27662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:41009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:34709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.1
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:27:39.216925 27662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:39.218081 27662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:39.228015 27668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:39.228032 27670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:39.228112 27662 server_base.cc:1034] running on GCE node
W20260923 17:27:39.228044 27667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:39.228734 27662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:39.229523 27662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:39.230711 27662 hybrid_clock.cc:648] HybridClock initialized: now 1790184459230696 us; error 35 us; skew 500 ppm
Sep 23 17:27:39 dist-test-slave-ptl9 krb5kdc[27555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184459, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:39.233517 27662 init.cc:377] Logged in from keytab as kudu/127.11.229.1@KRBTEST.COM (short username kudu)
I20260923 17:27:39.234407 27662 webserver.cc:460] Webserver started at http://127.11.229.1:36969/ using document root <none> and password file <none>
I20260923 17:27:39.235036 27662 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:39.235133 27662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:39.235373 27662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:27:39.237690 27662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/data/instance:
uuid: "cd15e54eada447f5b9e949c46b37a1b0"
format_stamp: "Formatted at 2026-09-23 17:27:39 on dist-test-slave-ptl9"
server_key: "1d28c559d8bb506051c0f17f650cb8bd"
server_key_iv: "19035947c5e6b01ded0a1325a0ac2a2f"
server_key_version: "encryptionkey@0"
I20260923 17:27:39.238318 27662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance:
uuid: "cd15e54eada447f5b9e949c46b37a1b0"
format_stamp: "Formatted at 2026-09-23 17:27:39 on dist-test-slave-ptl9"
server_key: "1d28c559d8bb506051c0f17f650cb8bd"
server_key_iv: "19035947c5e6b01ded0a1325a0ac2a2f"
server_key_version: "encryptionkey@0"
I20260923 17:27:39.242403 27662 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:27:39.244944 27677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:39.246335 27662 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:27:39.246479 27662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/wal
uuid: "cd15e54eada447f5b9e949c46b37a1b0"
format_stamp: "Formatted at 2026-09-23 17:27:39 on dist-test-slave-ptl9"
server_key: "1d28c559d8bb506051c0f17f650cb8bd"
server_key_iv: "19035947c5e6b01ded0a1325a0ac2a2f"
server_key_version: "encryptionkey@0"
I20260923 17:27:39.246634 27662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:39.272789 27662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:39.275950 27662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:39.276113 27662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:39.276711 27662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:27:39.277642 27662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:27:39.277698 27662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:39.277741 27662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:27:39.277760 27662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:39.287539 27662 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.1:45881
I20260923 17:27:39.287561 27790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.1:45881 every 8 connection(s)
I20260923 17:27:39.288551 27662 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
I20260923 17:27:39.288916 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 27662
I20260923 17:27:39.289070 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance
I20260923 17:27:39.289465 12180 external_mini_cluster.cc:1428] Setting key 3702ef73f2917a4a7beadb554f269297
Sep 23 17:27:39 dist-test-slave-ptl9 krb5kdc[27555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184459, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.229.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.2@KRBTEST.COM" created.
I20260923 17:27:39.302772 27639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:39.290984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:49177 (local address 127.11.229.62:34709)
0923 17:27:39.291158 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:39.291163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:39.292127 (+ 964us) server_negotiation.cc:407] Connection header received
0923 17:27:39.293066 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:39.293069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:39.293139 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:39.293244 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:39.294475 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:39.295037 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:39.296178 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:39.296419 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:39.296539 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:39.299046 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:39.299049 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:39.299051 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:39.299074 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:39.300592 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:39.301149 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:39.301153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:39.301155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:39.301242 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:39.301610 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:39.301614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:39.301616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:39.301820 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:39.301930 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:39.302447 (+ 517us) server_negotiation.cc:299] Negotiation successful
0923 17:27:39.302572 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260923 17:27:39.303668 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:39.291233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.1:49177)
0923 17:27:39.291977 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:39.292021 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:39.292849 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:39.293454 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:39.293461 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:39.293830 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:39.294324 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:39.294338 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:39.295180 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:39.295183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:39.296049 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:39.296059 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:39.296582 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:39.296586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:39.296758 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:39.297418 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:39.297433 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:39.298879 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:39.300754 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:39.300760 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:39.300776 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:39.301041 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:39.301358 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:39.301362 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:39.301365 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:39.301513 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:39.301932 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:39.301939 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:39.302204 (+ 265us) client_negotiation.cc:769] Sending connection context
0923 17:27:39.302394 (+ 190us) client_negotiation.cc:240] Negotiation successful
0923 17:27:39.302739 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":146,"threads_started":1}
I20260923 17:27:39.304934 27791 heartbeater.cc:347] Connected to a master server at 127.11.229.62:34709
I20260923 17:27:39.305207 27791 heartbeater.cc:464] Registering TS with master...
I20260923 17:27:39.305776 27791 heartbeater.cc:511] Master 127.11.229.62:34709 requested a full tablet report, sending...
I20260923 17:27:39.307348 27600 ts_manager.cc:194] Registered new tserver with Master: cd15e54eada447f5b9e949c46b37a1b0 (127.11.229.1:45881)
I20260923 17:27:39.308517 27600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.1@KRBTEST.COM'} at 127.11.229.1:49177
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab.
Entry for principal kudu/127.11.229.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab.
I20260923 17:27:39.316428 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.2:0
--local_ip_for_outbound_sockets=127.11.229.2
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:34709
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:41009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:27:39.429087 27796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:39.429392 27796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:39.429440 27796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:39.433566 27796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:39.433619 27796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:39.433676 27796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.2
I20260923 17:27:39.438853 27796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:41009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:34709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.2
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:27:39.439858 27796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:39.440821 27796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:39.450484 27801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:39.450511 27802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:39.450613 27796 server_base.cc:1034] running on GCE node
W20260923 17:27:39.450532 27804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:39.451290 27796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:39.451889 27796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:39.453104 27796 hybrid_clock.cc:648] HybridClock initialized: now 1790184459453048 us; error 40 us; skew 500 ppm
Sep 23 17:27:39 dist-test-slave-ptl9 krb5kdc[27555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184459, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:39.456264 27796 init.cc:377] Logged in from keytab as kudu/127.11.229.2@KRBTEST.COM (short username kudu)
I20260923 17:27:39.457208 27796 webserver.cc:460] Webserver started at http://127.11.229.2:42393/ using document root <none> and password file <none>
I20260923 17:27:39.457854 27796 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:39.457922 27796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:39.458113 27796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:27:39.460395 27796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/data/instance:
uuid: "97d1965102a7423f87a0de343bdbd507"
format_stamp: "Formatted at 2026-09-23 17:27:39 on dist-test-slave-ptl9"
server_key: "46a87e6b4f1a26a1f002e65875cd1473"
server_key_iv: "d6a284028d11b851ddcc93c3be748ccb"
server_key_version: "encryptionkey@0"
I20260923 17:27:39.461092 27796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance:
uuid: "97d1965102a7423f87a0de343bdbd507"
format_stamp: "Formatted at 2026-09-23 17:27:39 on dist-test-slave-ptl9"
server_key: "46a87e6b4f1a26a1f002e65875cd1473"
server_key_iv: "d6a284028d11b851ddcc93c3be748ccb"
server_key_version: "encryptionkey@0"
I20260923 17:27:39.465369 27796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:27:39.467730 27811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:39.468951 27796 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:27:39.469087 27796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/wal
uuid: "97d1965102a7423f87a0de343bdbd507"
format_stamp: "Formatted at 2026-09-23 17:27:39 on dist-test-slave-ptl9"
server_key: "46a87e6b4f1a26a1f002e65875cd1473"
server_key_iv: "d6a284028d11b851ddcc93c3be748ccb"
server_key_version: "encryptionkey@0"
I20260923 17:27:39.469228 27796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:39.497525 27796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:39.500900 27796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:39.501114 27796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:39.501791 27796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:27:39.502638 27796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:27:39.502708 27796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:39.502776 27796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:27:39.502820 27796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:39.511701 27796 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.2:42517
I20260923 17:27:39.511730 27924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.2:42517 every 8 connection(s)
I20260923 17:27:39.512636 27796 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
I20260923 17:27:39.513180 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 27796
I20260923 17:27:39.513342 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance
I20260923 17:27:39.513674 12180 external_mini_cluster.cc:1428] Setting key 6c82544165300c8bda28cc725fe73e59
Sep 23 17:27:39 dist-test-slave-ptl9 krb5kdc[27555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184459, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.229.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.3@KRBTEST.COM" created.
I20260923 17:27:39.528205 27639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:39.514617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:46993 (local address 127.11.229.62:34709)
0923 17:27:39.514776 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:39.514781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:39.516174 (+ 1393us) server_negotiation.cc:407] Connection header received
0923 17:27:39.517516 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:39.517520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:39.517597 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:39.517697 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:39.518861 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:39.519408 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:39.521162 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:39.521468 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:39.521561 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:39.524351 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:39.524355 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:39.524357 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:39.524382 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:39.525771 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:39.526335 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:39.526338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:39.526340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:39.526442 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:39.526913 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:39.526916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:39.526917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:39.527097 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:39.527189 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:39.527849 (+ 660us) server_negotiation.cc:299] Negotiation successful
0923 17:27:39.527962 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 17:27:39.529349 27927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:39.515176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.2:46993)
0923 17:27:39.515999 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:39.516058 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:39.517301 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:39.517883 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:39.517890 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:39.518234 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:39.518716 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:39.518729 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:39.519532 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:39.519535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:39.521004 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:39.521020 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:39.521631 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:39.521638 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:39.521865 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:39.522496 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:39.522515 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:39.524170 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:39.525908 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:39.525914 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:39.525941 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:39.526197 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:39.526592 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:39.526596 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:39.526599 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:39.526803 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:39.527216 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:39.527226 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:39.527558 (+ 332us) client_negotiation.cc:769] Sending connection context
0923 17:27:39.527780 (+ 222us) client_negotiation.cc:240] Negotiation successful
0923 17:27:39.528205 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":118,"threads_started":1}
I20260923 17:27:39.530645 27925 heartbeater.cc:347] Connected to a master server at 127.11.229.62:34709
I20260923 17:27:39.530908 27925 heartbeater.cc:464] Registering TS with master...
I20260923 17:27:39.531474 27925 heartbeater.cc:511] Master 127.11.229.62:34709 requested a full tablet report, sending...
I20260923 17:27:39.532552 27600 ts_manager.cc:194] Registered new tserver with Master: 97d1965102a7423f87a0de343bdbd507 (127.11.229.2:42517)
I20260923 17:27:39.533135 27600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.2@KRBTEST.COM'} at 127.11.229.2:46993
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab.
Entry for principal kudu/127.11.229.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab.
I20260923 17:27:39.544183 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.3:0
--local_ip_for_outbound_sockets=127.11.229.3
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:34709
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:41009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:27:39.655227 27930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:27:39.655519 27930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:27:39.655596 27930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:27:39.659583 27930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:27:39.659677 27930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:27:39.659771 27930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.3
I20260923 17:27:39.665482 27930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:41009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:34709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.3
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:27:39.666760 27930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:27:39.667922 27930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:27:39.678454 27935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:27:39.678454 27936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:39.678565 27930 server_base.cc:1034] running on GCE node
W20260923 17:27:39.678493 27938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:27:39.679185 27930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:27:39.679813 27930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:27:39.680996 27930 hybrid_clock.cc:648] HybridClock initialized: now 1790184459680965 us; error 51 us; skew 500 ppm
Sep 23 17:27:39 dist-test-slave-ptl9 krb5kdc[27555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184459, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:27:39.683758 27930 init.cc:377] Logged in from keytab as kudu/127.11.229.3@KRBTEST.COM (short username kudu)
I20260923 17:27:39.684525 27930 webserver.cc:460] Webserver started at http://127.11.229.3:33533/ using document root <none> and password file <none>
I20260923 17:27:39.685039 27930 fs_manager.cc:359] Metadata directory not provided
I20260923 17:27:39.685093 27930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:27:39.685303 27930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:27:39.687287 27930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/data/instance:
uuid: "76d05fb754494f3582f4d96f1d3dd5f0"
format_stamp: "Formatted at 2026-09-23 17:27:39 on dist-test-slave-ptl9"
server_key: "986939a2dd0d35b6bff2290816021f51"
server_key_iv: "26c775f349b525de743936e442fc1b76"
server_key_version: "encryptionkey@0"
I20260923 17:27:39.687795 27930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance:
uuid: "76d05fb754494f3582f4d96f1d3dd5f0"
format_stamp: "Formatted at 2026-09-23 17:27:39 on dist-test-slave-ptl9"
server_key: "986939a2dd0d35b6bff2290816021f51"
server_key_iv: "26c775f349b525de743936e442fc1b76"
server_key_version: "encryptionkey@0"
I20260923 17:27:39.691495 27930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 17:27:39.693879 27945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:39.695111 27930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:27:39.695212 27930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/wal
uuid: "76d05fb754494f3582f4d96f1d3dd5f0"
format_stamp: "Formatted at 2026-09-23 17:27:39 on dist-test-slave-ptl9"
server_key: "986939a2dd0d35b6bff2290816021f51"
server_key_iv: "26c775f349b525de743936e442fc1b76"
server_key_version: "encryptionkey@0"
I20260923 17:27:39.695298 27930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:27:39.713069 27930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:27:39.716495 27930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:27:39.716727 27930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:27:39.717373 27930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:27:39.718212 27930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:27:39.718283 27930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:39.718357 27930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:27:39.718412 27930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:27:39.726976 27930 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.3:34709
I20260923 17:27:39.727048 28058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.3:34709 every 8 connection(s)
I20260923 17:27:39.727933 27930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
I20260923 17:27:39.730500 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 27930
I20260923 17:27:39.730655 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance
I20260923 17:27:39.731014 12180 external_mini_cluster.cc:1428] Setting key b2431388f7271f9c95d803223c28357b
Sep 23 17:27:39 dist-test-slave-ptl9 krb5kdc[27555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184459, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:27:39.741582 27639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:39.729944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:50513 (local address 127.11.229.62:34709)
0923 17:27:39.730081 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:39.730085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:39.731447 (+ 1362us) server_negotiation.cc:407] Connection header received
0923 17:27:39.732508 (+ 1061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:39.732512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:39.732569 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:39.732636 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:39.733989 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:39.734523 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:39.735610 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:39.735826 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:39.735902 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:39.738378 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:39.738381 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:39.738383 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:39.738408 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:39.739608 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:39.740072 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:39.740075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:39.740076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:39.740147 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:39.740429 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:39.740431 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:39.740432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:39.740611 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:39.740685 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:39.741209 (+ 524us) server_negotiation.cc:299] Negotiation successful
0923 17:27:39.741392 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 17:27:39.742429 28061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:39.730454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.3:50513)
0923 17:27:39.731264 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:39.731327 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:39.732287 (+ 960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:39.732814 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:39.732822 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:39.733377 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:27:39.733860 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:39.733873 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:39.734627 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:39.734629 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:39.735509 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:39.735517 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:39.735957 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:39.735960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:39.736112 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:39.736745 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:27:39.736762 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:27:39.738225 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:27:39.739725 (+ 1500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:39.739730 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:39.739742 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:39.739987 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:39.740224 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:27:39.740227 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:27:39.740228 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:27:39.740353 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:27:39.740732 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:27:39.740739 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:27:39.740993 (+ 254us) client_negotiation.cc:769] Sending connection context
0923 17:27:39.741168 (+ 175us) client_negotiation.cc:240] Negotiation successful
0923 17:27:39.741586 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":161,"threads_started":1}
I20260923 17:27:39.743686 28059 heartbeater.cc:347] Connected to a master server at 127.11.229.62:34709
I20260923 17:27:39.743942 28059 heartbeater.cc:464] Registering TS with master...
I20260923 17:27:39.744611 28059 heartbeater.cc:511] Master 127.11.229.62:34709 requested a full tablet report, sending...
I20260923 17:27:39.745711 27600 ts_manager.cc:194] Registered new tserver with Master: 76d05fb754494f3582f4d96f1d3dd5f0 (127.11.229.3:34709)
I20260923 17:27:39.746246 27600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.3@KRBTEST.COM'} at 127.11.229.3:50513
I20260923 17:27:39.756660 12180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:27:39.765553 27639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:39.758722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43426 (local address 127.11.229.62:34709)
0923 17:27:39.758854 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:39.758858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:39.759014 (+ 156us) server_negotiation.cc:407] Connection header received
0923 17:27:39.759113 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:39.759115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:39.759154 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:39.759223 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:27:39.759839 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:39.760355 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:39.761157 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:39.761452 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:39.761528 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:39.762752 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:27:39.762756 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:27:39.762758 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:27:39.762780 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:27:39.764146 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:39.764519 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:39.764522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:39.764523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:39.764592 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:27:39.764809 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:27:39.764811 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:27:39.764812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:27:39.765010 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:27:39.765106 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:39.765344 (+ 238us) server_negotiation.cc:299] Negotiation successful
0923 17:27:39.765437 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 17:27:39.768255 27600 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43426:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:27:39.771422 27600 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:27:39.785804 28070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:39.779926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:34709 (local address 127.0.0.1:40238)
0923 17:27:39.780314 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:39.780369 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:39.780511 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:39.781356 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:39.781361 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:39.781386 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:27:39.781528 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:39.781536 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:39.782935 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:39.782938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:39.784345 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:39.784355 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:39.785224 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:39.785227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:39.785375 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:39.785420 (+ 45us) client_negotiation.cc:769] Sending connection context
0923 17:27:39.785534 (+ 114us) client_negotiation.cc:240] Negotiation successful
0923 17:27:39.785646 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":103,"threads_started":1}
I20260923 17:27:39.785832 28071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:39.780105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40238 (local address 127.11.229.3:34709)
0923 17:27:39.780495 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:39.780507 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:39.780532 (+ 25us) server_negotiation.cc:407] Connection header received
0923 17:27:39.780729 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:39.780735 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:39.780966 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:39.781132 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:27:39.781658 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:39.782791 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:39.784476 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:39.785095 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:39.785217 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:39.785394 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:39.785577 (+ 183us) server_negotiation.cc:299] Negotiation successful
0923 17:27:39.785682 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":95,"threads_started":1}
I20260923 17:27:39.786774 28072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:39.780284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:45881 (local address 127.0.0.1:45758)
0923 17:27:39.780863 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:39.780876 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:39.780953 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:39.782215 (+ 1262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:39.782218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:39.782237 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:27:39.782372 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:39.782378 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:39.783872 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:39.783875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:39.784971 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:39.784981 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:39.786202 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:39.786206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:39.786398 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:39.786455 (+ 57us) client_negotiation.cc:769] Sending connection context
0923 17:27:39.786530 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 17:27:39.786646 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":84,"thread_start_us":103,"threads_started":1}
I20260923 17:27:39.786914 28075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:39.781043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45758 (local address 127.11.229.1:45881)
0923 17:27:39.781692 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:39.781712 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:39.781747 (+ 35us) server_negotiation.cc:407] Connection header received
0923 17:27:39.781833 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:39.781838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:39.782036 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:39.782512 (+ 476us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:27:39.782655 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:39.783713 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:39.785114 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:39.786078 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:39.786180 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:39.786328 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:39.786660 (+ 332us) server_negotiation.cc:299] Negotiation successful
0923 17:27:39.786756 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":118,"threads_started":1}
I20260923 17:27:39.787144 28073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:39.780699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:42517 (local address 127.0.0.1:56194)
0923 17:27:39.781566 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:39.781580 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:39.781680 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:39.782114 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:39.782117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:39.782134 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:27:39.782233 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:39.782240 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:39.783635 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:39.783639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:39.784906 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:39.784917 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:39.786202 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:39.786206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:39.786426 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:39.786459 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 17:27:39.786530 (+ 71us) client_negotiation.cc:240] Negotiation successful
0923 17:27:39.787016 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806,"mutex_wait_us":106,"thread_start_us":83,"threads_started":1}
I20260923 17:27:39.787886 28074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:39.781059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56194 (local address 127.11.229.2:42517)
0923 17:27:39.781503 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:39.781522 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:39.781678 (+ 156us) server_negotiation.cc:407] Connection header received
0923 17:27:39.781796 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:39.781800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:39.781968 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:39.782135 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:27:39.782369 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:39.783374 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:39.785048 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:39.785904 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:39.786008 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:39.786125 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:39.786562 (+ 437us) server_negotiation.cc:299] Negotiation successful
0923 17:27:39.786652 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":89,"threads_started":1}
I20260923 17:27:39.789443 27993 tablet_service.cc:1467] Processing CreateTablet for tablet d61349c609ee440fa0ff7eebcdecbc18 (DEFAULT_TABLE table=test-table [id=38eb2a456b864c2da48a0aa3ec972f55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:27:39.789681 27725 tablet_service.cc:1467] Processing CreateTablet for tablet d61349c609ee440fa0ff7eebcdecbc18 (DEFAULT_TABLE table=test-table [id=38eb2a456b864c2da48a0aa3ec972f55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:27:39.790091 27859 tablet_service.cc:1467] Processing CreateTablet for tablet d61349c609ee440fa0ff7eebcdecbc18 (DEFAULT_TABLE table=test-table [id=38eb2a456b864c2da48a0aa3ec972f55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:27:39.790632 27993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d61349c609ee440fa0ff7eebcdecbc18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:27:39.790632 27725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d61349c609ee440fa0ff7eebcdecbc18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:27:39.791106 27859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d61349c609ee440fa0ff7eebcdecbc18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:27:39.797186 28076 tablet_bootstrap.cc:492] T d61349c609ee440fa0ff7eebcdecbc18 P 97d1965102a7423f87a0de343bdbd507: Bootstrap starting.
I20260923 17:27:39.798552 28077 tablet_bootstrap.cc:492] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0: Bootstrap starting.
I20260923 17:27:39.800339 28076 tablet_bootstrap.cc:654] T d61349c609ee440fa0ff7eebcdecbc18 P 97d1965102a7423f87a0de343bdbd507: Neither blocks nor log segments found. Creating new log.
I20260923 17:27:39.800341 28077 tablet_bootstrap.cc:654] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0: Neither blocks nor log segments found. Creating new log.
I20260923 17:27:39.801110 28076 log.cc:824] T d61349c609ee440fa0ff7eebcdecbc18 P 97d1965102a7423f87a0de343bdbd507: Log is configured to *not* fsync() on all Append() calls
I20260923 17:27:39.801469 28077 log.cc:824] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0: Log is configured to *not* fsync() on all Append() calls
I20260923 17:27:39.802160 28078 tablet_bootstrap.cc:492] T d61349c609ee440fa0ff7eebcdecbc18 P 76d05fb754494f3582f4d96f1d3dd5f0: Bootstrap starting.
I20260923 17:27:39.803085 28076 tablet_bootstrap.cc:492] T d61349c609ee440fa0ff7eebcdecbc18 P 97d1965102a7423f87a0de343bdbd507: No bootstrap required, opened a new log
I20260923 17:27:39.803285 28076 ts_tablet_manager.cc:1397] T d61349c609ee440fa0ff7eebcdecbc18 P 97d1965102a7423f87a0de343bdbd507: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260923 17:27:39.804056 28078 tablet_bootstrap.cc:654] T d61349c609ee440fa0ff7eebcdecbc18 P 76d05fb754494f3582f4d96f1d3dd5f0: Neither blocks nor log segments found. Creating new log.
I20260923 17:27:39.805158 28078 log.cc:824] T d61349c609ee440fa0ff7eebcdecbc18 P 76d05fb754494f3582f4d96f1d3dd5f0: Log is configured to *not* fsync() on all Append() calls
I20260923 17:27:39.806509 28076 raft_consensus.cc:348] T d61349c609ee440fa0ff7eebcdecbc18 P 97d1965102a7423f87a0de343bdbd507 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d05fb754494f3582f4d96f1d3dd5f0" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34709 } } peers { permanent_uuid: "97d1965102a7423f87a0de343bdbd507" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42517 } } peers { permanent_uuid: "cd15e54eada447f5b9e949c46b37a1b0" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45881 } }
I20260923 17:27:39.806731 28076 raft_consensus.cc:374] T d61349c609ee440fa0ff7eebcdecbc18 P 97d1965102a7423f87a0de343bdbd507 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:27:39.806764 28076 raft_consensus.cc:729] T d61349c609ee440fa0ff7eebcdecbc18 P 97d1965102a7423f87a0de343bdbd507 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97d1965102a7423f87a0de343bdbd507, State: Initialized, Role: FOLLOWER
I20260923 17:27:39.807132 28078 tablet_bootstrap.cc:492] T d61349c609ee440fa0ff7eebcdecbc18 P 76d05fb754494f3582f4d96f1d3dd5f0: No bootstrap required, opened a new log
I20260923 17:27:39.807192 28076 consensus_queue.cc:260] T d61349c609ee440fa0ff7eebcdecbc18 P 97d1965102a7423f87a0de343bdbd507 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d05fb754494f3582f4d96f1d3dd5f0" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34709 } } peers { permanent_uuid: "97d1965102a7423f87a0de343bdbd507" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42517 } } peers { permanent_uuid: "cd15e54eada447f5b9e949c46b37a1b0" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45881 } }
I20260923 17:27:39.807358 28078 ts_tablet_manager.cc:1397] T d61349c609ee440fa0ff7eebcdecbc18 P 76d05fb754494f3582f4d96f1d3dd5f0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 17:27:39.807919 27925 heartbeater.cc:503] Master 127.11.229.62:34709 was elected leader, sending a full tablet report...
I20260923 17:27:39.808220 28076 ts_tablet_manager.cc:1428] T d61349c609ee440fa0ff7eebcdecbc18 P 97d1965102a7423f87a0de343bdbd507: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260923 17:27:39.810500 28077 tablet_bootstrap.cc:492] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0: No bootstrap required, opened a new log
I20260923 17:27:39.810726 28077 ts_tablet_manager.cc:1397] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260923 17:27:39.810688 28078 raft_consensus.cc:348] T d61349c609ee440fa0ff7eebcdecbc18 P 76d05fb754494f3582f4d96f1d3dd5f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d05fb754494f3582f4d96f1d3dd5f0" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34709 } } peers { permanent_uuid: "97d1965102a7423f87a0de343bdbd507" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42517 } } peers { permanent_uuid: "cd15e54eada447f5b9e949c46b37a1b0" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45881 } }
I20260923 17:27:39.810968 28078 raft_consensus.cc:374] T d61349c609ee440fa0ff7eebcdecbc18 P 76d05fb754494f3582f4d96f1d3dd5f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:27:39.811038 28078 raft_consensus.cc:729] T d61349c609ee440fa0ff7eebcdecbc18 P 76d05fb754494f3582f4d96f1d3dd5f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d05fb754494f3582f4d96f1d3dd5f0, State: Initialized, Role: FOLLOWER
I20260923 17:27:39.811446 28078 consensus_queue.cc:260] T d61349c609ee440fa0ff7eebcdecbc18 P 76d05fb754494f3582f4d96f1d3dd5f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d05fb754494f3582f4d96f1d3dd5f0" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34709 } } peers { permanent_uuid: "97d1965102a7423f87a0de343bdbd507" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42517 } } peers { permanent_uuid: "cd15e54eada447f5b9e949c46b37a1b0" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45881 } }
I20260923 17:27:39.812100 28059 heartbeater.cc:503] Master 127.11.229.62:34709 was elected leader, sending a full tablet report...
I20260923 17:27:39.812493 28078 ts_tablet_manager.cc:1428] T d61349c609ee440fa0ff7eebcdecbc18 P 76d05fb754494f3582f4d96f1d3dd5f0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260923 17:27:39.814357 28077 raft_consensus.cc:348] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d05fb754494f3582f4d96f1d3dd5f0" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34709 } } peers { permanent_uuid: "97d1965102a7423f87a0de343bdbd507" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42517 } } peers { permanent_uuid: "cd15e54eada447f5b9e949c46b37a1b0" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45881 } }
I20260923 17:27:39.814608 28077 raft_consensus.cc:374] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:27:39.814648 28077 raft_consensus.cc:729] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd15e54eada447f5b9e949c46b37a1b0, State: Initialized, Role: FOLLOWER
I20260923 17:27:39.815105 28077 consensus_queue.cc:260] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d05fb754494f3582f4d96f1d3dd5f0" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34709 } } peers { permanent_uuid: "97d1965102a7423f87a0de343bdbd507" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42517 } } peers { permanent_uuid: "cd15e54eada447f5b9e949c46b37a1b0" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45881 } }
I20260923 17:27:39.815763 27791 heartbeater.cc:503] Master 127.11.229.62:34709 was elected leader, sending a full tablet report...
I20260923 17:27:39.815973 28077 ts_tablet_manager.cc:1428] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260923 17:27:39.978922 28060 tablet.cc:2369] T d61349c609ee440fa0ff7eebcdecbc18 P 76d05fb754494f3582f4d96f1d3dd5f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:27:40.007043 27631 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.11.229.62:34709
W20260923 17:27:40.013823 27926 tablet.cc:2369] T d61349c609ee440fa0ff7eebcdecbc18 P 97d1965102a7423f87a0de343bdbd507: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:27:40.040197 27792 tablet.cc:2369] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:27:40.051956 28084 raft_consensus.cc:482] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:27:40.052201 28084 raft_consensus.cc:504] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d05fb754494f3582f4d96f1d3dd5f0" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34709 } } peers { permanent_uuid: "97d1965102a7423f87a0de343bdbd507" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42517 } } peers { permanent_uuid: "cd15e54eada447f5b9e949c46b37a1b0" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45881 } }
I20260923 17:27:40.053474 28084 leader_election.cc:288] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76d05fb754494f3582f4d96f1d3dd5f0 (127.11.229.3:34709), 97d1965102a7423f87a0de343bdbd507 (127.11.229.2:42517)
I20260923 17:27:40.058681 28071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:40.053857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:42631 (local address 127.11.229.3:34709)
0923 17:27:40.053984 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:40.053987 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:40.054212 (+ 225us) server_negotiation.cc:407] Connection header received
0923 17:27:40.054374 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:40.054377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:40.054432 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:40.054527 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:27:40.054848 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:40.055636 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:40.057347 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:40.057994 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:40.058107 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:40.058160 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:40.058461 (+ 301us) server_negotiation.cc:299] Negotiation successful
0923 17:27:40.058532 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 17:27:40.058668 28074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:40.054335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:37313 (local address 127.11.229.2:42517)
0923 17:27:40.054460 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:40.054466 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:40.054668 (+ 202us) server_negotiation.cc:407] Connection header received
0923 17:27:40.054752 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:40.054754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:40.054815 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:40.054888 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:27:40.055109 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:40.055805 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:40.057334 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:40.057985 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:40.058107 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:40.058160 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:40.058461 (+ 301us) server_negotiation.cc:299] Negotiation successful
0923 17:27:40.058532 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 17:27:40.058741 28085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:40.053755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:34709 (local address 127.11.229.1:42631)
0923 17:27:40.054080 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:40.054097 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:40.054233 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:40.054554 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:40.054557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:40.054578 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:27:40.054738 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:40.054744 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:40.055749 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:40.055752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:40.057186 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:40.057197 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:40.058128 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:40.058132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:40.058310 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:40.058354 (+ 44us) client_negotiation.cc:769] Sending connection context
0923 17:27:40.058432 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 17:27:40.058552 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":117,"thread_start_us":121,"threads_started":1}
I20260923 17:27:40.058741 28086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:40.054192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:42517 (local address 127.11.229.1:37313)
0923 17:27:40.054569 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:40.054581 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 17:27:40.054666 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:27:40.054904 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:27:40.054906 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:27:40.054922 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:27:40.055011 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:40.055015 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:40.055901 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:40.055904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:40.057186 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:27:40.057197 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:27:40.058128 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:27:40.058132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:27:40.058304 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:40.058351 (+ 47us) client_negotiation.cc:769] Sending connection context
0923 17:27:40.058432 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 17:27:40.058552 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":173,"thread_start_us":91,"threads_started":1}
I20260923 17:27:40.059523 27879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61349c609ee440fa0ff7eebcdecbc18" candidate_uuid: "cd15e54eada447f5b9e949c46b37a1b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97d1965102a7423f87a0de343bdbd507" is_pre_election: true
I20260923 17:27:40.059525 28013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61349c609ee440fa0ff7eebcdecbc18" candidate_uuid: "cd15e54eada447f5b9e949c46b37a1b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d05fb754494f3582f4d96f1d3dd5f0" is_pre_election: true
I20260923 17:27:40.059862 28013 raft_consensus.cc:2454] T d61349c609ee440fa0ff7eebcdecbc18 P 76d05fb754494f3582f4d96f1d3dd5f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd15e54eada447f5b9e949c46b37a1b0 in term 0.
I20260923 17:27:40.059862 27879 raft_consensus.cc:2454] T d61349c609ee440fa0ff7eebcdecbc18 P 97d1965102a7423f87a0de343bdbd507 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd15e54eada447f5b9e949c46b37a1b0 in term 0.
I20260923 17:27:40.060353 27678 leader_election.cc:302] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d05fb754494f3582f4d96f1d3dd5f0, cd15e54eada447f5b9e949c46b37a1b0; no voters:
I20260923 17:27:40.060585 28084 raft_consensus.cc:2781] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:27:40.060699 28084 raft_consensus.cc:482] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:27:40.060776 28084 raft_consensus.cc:3037] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:27:40.061991 28084 raft_consensus.cc:504] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d05fb754494f3582f4d96f1d3dd5f0" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34709 } } peers { permanent_uuid: "97d1965102a7423f87a0de343bdbd507" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42517 } } peers { permanent_uuid: "cd15e54eada447f5b9e949c46b37a1b0" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45881 } }
I20260923 17:27:40.062342 28084 leader_election.cc:288] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0 [CANDIDATE]: Term 1 election: Requested vote from peers 76d05fb754494f3582f4d96f1d3dd5f0 (127.11.229.3:34709), 97d1965102a7423f87a0de343bdbd507 (127.11.229.2:42517)
I20260923 17:27:40.062717 28013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61349c609ee440fa0ff7eebcdecbc18" candidate_uuid: "cd15e54eada447f5b9e949c46b37a1b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d05fb754494f3582f4d96f1d3dd5f0"
I20260923 17:27:40.062800 27879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61349c609ee440fa0ff7eebcdecbc18" candidate_uuid: "cd15e54eada447f5b9e949c46b37a1b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97d1965102a7423f87a0de343bdbd507"
I20260923 17:27:40.062853 28013 raft_consensus.cc:3037] T d61349c609ee440fa0ff7eebcdecbc18 P 76d05fb754494f3582f4d96f1d3dd5f0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:27:40.062968 27879 raft_consensus.cc:3037] T d61349c609ee440fa0ff7eebcdecbc18 P 97d1965102a7423f87a0de343bdbd507 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:27:40.063905 28013 raft_consensus.cc:2454] T d61349c609ee440fa0ff7eebcdecbc18 P 76d05fb754494f3582f4d96f1d3dd5f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd15e54eada447f5b9e949c46b37a1b0 in term 1.
I20260923 17:27:40.064034 27879 raft_consensus.cc:2454] T d61349c609ee440fa0ff7eebcdecbc18 P 97d1965102a7423f87a0de343bdbd507 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd15e54eada447f5b9e949c46b37a1b0 in term 1.
I20260923 17:27:40.064225 27678 leader_election.cc:302] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d05fb754494f3582f4d96f1d3dd5f0, cd15e54eada447f5b9e949c46b37a1b0; no voters:
I20260923 17:27:40.064429 28084 raft_consensus.cc:2781] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:27:40.064651 28084 raft_consensus.cc:686] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0 [term 1 LEADER]: Becoming Leader. State: Replica: cd15e54eada447f5b9e949c46b37a1b0, State: Running, Role: LEADER
I20260923 17:27:40.065028 28084 consensus_queue.cc:237] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d05fb754494f3582f4d96f1d3dd5f0" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34709 } } peers { permanent_uuid: "97d1965102a7423f87a0de343bdbd507" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42517 } } peers { permanent_uuid: "cd15e54eada447f5b9e949c46b37a1b0" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45881 } }
I20260923 17:27:40.068297 27600 catalog_manager.cc:5495] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd15e54eada447f5b9e949c46b37a1b0 (127.11.229.1). New cstate: current_term: 1 leader_uuid: "cd15e54eada447f5b9e949c46b37a1b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d05fb754494f3582f4d96f1d3dd5f0" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97d1965102a7423f87a0de343bdbd507" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd15e54eada447f5b9e949c46b37a1b0" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45881 } health_report { overall_health: HEALTHY } } }
I20260923 17:27:40.148962 28075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:40.144819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45772 (local address 127.11.229.1:45881)
0923 17:27:40.144975 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0923 17:27:40.144982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:27:40.145000 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:27:40.145136 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:27:40.145139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:27:40.145205 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:27:40.145394 (+ 189us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:27:40.145678 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:40.146298 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:40.147295 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:27:40.147543 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:27:40.147672 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:27:40.148026 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:27:40.148500 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:27:40.148652 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 17:27:40.148724 (+ 72us) server_negotiation.cc:299] Negotiation successful
0923 17:27:40.148790 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:27:40.155625 27879 raft_consensus.cc:1261] T d61349c609ee440fa0ff7eebcdecbc18 P 97d1965102a7423f87a0de343bdbd507 [term 1 FOLLOWER]: Refusing update from remote peer cd15e54eada447f5b9e949c46b37a1b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:27:40.155625 28013 raft_consensus.cc:1261] T d61349c609ee440fa0ff7eebcdecbc18 P 76d05fb754494f3582f4d96f1d3dd5f0 [term 1 FOLLOWER]: Refusing update from remote peer cd15e54eada447f5b9e949c46b37a1b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:27:40.156401 28087 consensus_queue.cc:1035] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d05fb754494f3582f4d96f1d3dd5f0" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:27:40.156635 28084 consensus_queue.cc:1035] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97d1965102a7423f87a0de343bdbd507" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:27:40.164565 28090 mvcc.cc:205] Tried to move back new op lower bound from 7332595548788772864 to 7332595548429471744. Current Snapshot: MvccSnapshot[applied={T|T < 7332595548788772864}]
I20260923 17:27:40.168257 28091 mvcc.cc:205] Tried to move back new op lower bound from 7332595548788772864 to 7332595548429471744. Current Snapshot: MvccSnapshot[applied={T|T < 7332595548788772864 or (T in {7332595548788772864})}]
I20260923 17:27:40.173971 27600 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:27:40.174149 27600 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:27:40.176659 27600 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e: Sending DeleteTablet for 3 replicas of tablet d61349c609ee440fa0ff7eebcdecbc18
I20260923 17:27:40.177474 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 27662
I20260923 17:27:40.177505 27725 tablet_service.cc:1514] Processing DeleteTablet for tablet d61349c609ee440fa0ff7eebcdecbc18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:27:40 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:45758
I20260923 17:27:40.178114 27993 tablet_service.cc:1514] Processing DeleteTablet for tablet d61349c609ee440fa0ff7eebcdecbc18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:27:40 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:40238
I20260923 17:27:40.178325 27859 tablet_service.cc:1514] Processing DeleteTablet for tablet d61349c609ee440fa0ff7eebcdecbc18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:27:40 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:56194
I20260923 17:27:40.178804 28101 tablet_replica.cc:330] T d61349c609ee440fa0ff7eebcdecbc18 P 97d1965102a7423f87a0de343bdbd507: stopping tablet replica
I20260923 17:27:40.179009 28100 tablet_replica.cc:330] T d61349c609ee440fa0ff7eebcdecbc18 P 76d05fb754494f3582f4d96f1d3dd5f0: stopping tablet replica
I20260923 17:27:40.179215 28101 raft_consensus.cc:2229] T d61349c609ee440fa0ff7eebcdecbc18 P 97d1965102a7423f87a0de343bdbd507 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:27:40.179322 28100 raft_consensus.cc:2229] T d61349c609ee440fa0ff7eebcdecbc18 P 76d05fb754494f3582f4d96f1d3dd5f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:27:40.179553 28101 raft_consensus.cc:2258] T d61349c609ee440fa0ff7eebcdecbc18 P 97d1965102a7423f87a0de343bdbd507 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:27:40.179591 28100 raft_consensus.cc:2258] T d61349c609ee440fa0ff7eebcdecbc18 P 76d05fb754494f3582f4d96f1d3dd5f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:27:40.181190 28101 ts_tablet_manager.cc:1905] T d61349c609ee440fa0ff7eebcdecbc18 P 97d1965102a7423f87a0de343bdbd507: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:27:40.182456 28100 ts_tablet_manager.cc:1905] T d61349c609ee440fa0ff7eebcdecbc18 P 76d05fb754494f3582f4d96f1d3dd5f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:27:40.184238 28101 ts_tablet_manager.cc:1918] T d61349c609ee440fa0ff7eebcdecbc18 P 97d1965102a7423f87a0de343bdbd507: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:27:40.184336 28101 log.cc:1196] T d61349c609ee440fa0ff7eebcdecbc18 P 97d1965102a7423f87a0de343bdbd507: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/wal/wals/d61349c609ee440fa0ff7eebcdecbc18
I20260923 17:27:40.184638 28101 ts_tablet_manager.cc:1939] T d61349c609ee440fa0ff7eebcdecbc18 P 97d1965102a7423f87a0de343bdbd507: Deleting consensus metadata
I20260923 17:27:40.185189 28100 ts_tablet_manager.cc:1918] T d61349c609ee440fa0ff7eebcdecbc18 P 76d05fb754494f3582f4d96f1d3dd5f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:27:40.185307 28100 log.cc:1196] T d61349c609ee440fa0ff7eebcdecbc18 P 76d05fb754494f3582f4d96f1d3dd5f0: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/wal/wals/d61349c609ee440fa0ff7eebcdecbc18
I20260923 17:27:40.185634 28100 ts_tablet_manager.cc:1939] T d61349c609ee440fa0ff7eebcdecbc18 P 76d05fb754494f3582f4d96f1d3dd5f0: Deleting consensus metadata
I20260923 17:27:40.185985 27584 catalog_manager.cc:4841] TS 97d1965102a7423f87a0de343bdbd507 (127.11.229.2:42517): tablet d61349c609ee440fa0ff7eebcdecbc18 (table test-table [id=38eb2a456b864c2da48a0aa3ec972f55]) successfully deleted
W20260923 17:27:40.186002 27586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.229.1:45881 (error 108)
I20260923 17:27:40.186404 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 27796
I20260923 17:27:40.187441 28073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:27:40.187144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:45881 (local address 127.0.0.1:45782)
0923 17:27:40.187289 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0923 17:27:40.187358 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.1:45881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260923 17:27:40.187583 27586 catalog_manager.cc:4584] TS cd15e54eada447f5b9e949c46b37a1b0 (127.11.229.1:45881): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d61349c609ee440fa0ff7eebcdecbc18: Network error: Client connection negotiation failed: client connection to 127.11.229.1:45881: connect: Connection refused (error 111)
I20260923 17:27:40.188248 27584 catalog_manager.cc:4841] TS 76d05fb754494f3582f4d96f1d3dd5f0 (127.11.229.3:34709): tablet d61349c609ee440fa0ff7eebcdecbc18 (table test-table [id=38eb2a456b864c2da48a0aa3ec972f55]) successfully deleted
I20260923 17:27:40.193082 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 27930
I20260923 17:27:40.198621 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 27568
I20260923 17:28:00.205601 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:34709
--webserver_interface=127.11.229.62
--webserver_port=38351
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:41009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:34709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:00.318151 28103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:00.318421 28103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:00.318495 28103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:00.322563 28103 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 17:28:00.322616 28103 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 17:28:00.322633 28103 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 17:28:00.322646 28103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:28:00.322705 28103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:00.322739 28103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:00.322775 28103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:28:00.322815 28103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:28:00.328179 28103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:41009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:34709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:34709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=38351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:00.329314 28103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:00.330391 28103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:00.344182 28112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:00.344296 28109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:00.344271 28103 server_base.cc:1034] running on GCE node
W20260923 17:28:00.344193 28110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:00.345098 28103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:00.346393 28103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:00.347559 28103 hybrid_clock.cc:648] HybridClock initialized: now 1790184480347545 us; error 37 us; skew 500 ppm
Sep 23 17:28:00 dist-test-slave-ptl9 krb5kdc[27555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184480, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:00.350531 28103 init.cc:377] Logged in from keytab as kudu/127.11.229.62@KRBTEST.COM (short username kudu)
I20260923 17:28:00.351523 28103 webserver.cc:460] Webserver started at http://127.11.229.62:38351/ using document root <none> and password file <none>
I20260923 17:28:00.352150 28103 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:00.352231 28103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:00.356554 28103 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:28:00.358980 28119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:00.360306 28103 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260923 17:28:00.360435 28103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "0aa575a5a84b41b194b099dc7e71f04e"
format_stamp: "Formatted at 2026-09-23 17:27:38 on dist-test-slave-ptl9"
server_key: "59c03647ad7833031233bee6ca0e32be"
server_key_iv: "a60248ed97d608168500c8060557ef64"
server_key_version: "encryptionkey@0"
I20260923 17:28:00.361032 28103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:00.392136 28103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:00.395447 28103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:00.395623 28103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:00.403402 28103 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:34709
I20260923 17:28:00.403714 28171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:34709 every 8 connection(s)
I20260923 17:28:00.404788 28103 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:28:00.409451 28172 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:28:00.411338 28172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e: Bootstrap starting.
I20260923 17:28:00.412429 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 28103
I20260923 17:28:00.412954 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.1:45881
--local_ip_for_outbound_sockets=127.11.229.1
--tserver_master_addrs=127.11.229.62:34709
--webserver_port=36969
--webserver_interface=127.11.229.1
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:41009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:28:00.430042 28172 log.cc:824] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:00.439184 28172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:28:00.439579 28172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e: Bootstrap complete.
I20260923 17:28:00.443768 28172 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa575a5a84b41b194b099dc7e71f04e" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34709 } }
I20260923 17:28:00.444175 28172 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aa575a5a84b41b194b099dc7e71f04e, State: Initialized, Role: FOLLOWER
I20260923 17:28:00.444679 28172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa575a5a84b41b194b099dc7e71f04e" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34709 } }
I20260923 17:28:00.444803 28172 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:28:00.444898 28172 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:28:00.445014 28172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [term 1 FOLLOWER]: Advancing to term 2
I20260923 17:28:00.447032 28172 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa575a5a84b41b194b099dc7e71f04e" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34709 } }
I20260923 17:28:00.447393 28172 leader_election.cc:302] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0aa575a5a84b41b194b099dc7e71f04e; no voters:
I20260923 17:28:00.447747 28172 leader_election.cc:288] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 17:28:00.447829 28176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [term 2 FOLLOWER]: Leader election won for term 2
I20260923 17:28:00.448077 28176 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [term 2 LEADER]: Becoming Leader. State: Replica: 0aa575a5a84b41b194b099dc7e71f04e, State: Running, Role: LEADER
I20260923 17:28:00.448441 28176 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa575a5a84b41b194b099dc7e71f04e" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34709 } }
I20260923 17:28:00.449093 28172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:28:00.450028 28178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0aa575a5a84b41b194b099dc7e71f04e. Latest consensus state: current_term: 2 leader_uuid: "0aa575a5a84b41b194b099dc7e71f04e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa575a5a84b41b194b099dc7e71f04e" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34709 } } }
I20260923 17:28:00.450151 28178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:00.450081 28177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0aa575a5a84b41b194b099dc7e71f04e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa575a5a84b41b194b099dc7e71f04e" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34709 } } }
I20260923 17:28:00.450208 28177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:00.454330 28191 catalog_manager.cc:1259] Loaded cluster ID: e7a40a4a7b144207bdc56630af64e984
I20260923 17:28:00.454412 28191 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e: loading cluster ID for follower catalog manager: success
I20260923 17:28:00.456212 28191 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e: acquiring CA information for follower catalog manager: success
I20260923 17:28:00.457697 28191 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 17:28:00.457856 28185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:28:00.460690 28185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:28:00.461159 28185 catalog_manager.cc:1259] Loaded cluster ID: e7a40a4a7b144207bdc56630af64e984
I20260923 17:28:00.461243 28185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:28:00.461961 28185 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:28:00.462438 28185 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e: Loaded TSK: 0
I20260923 17:28:00.486159 28185 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e: Generated new TSK 1
I20260923 17:28:00.486464 28185 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 17:28:00.537461 28174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:00.537729 28174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:00.537774 28174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:00.541894 28174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:00.541955 28174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:00.542043 28174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.1
I20260923 17:28:00.547421 28174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:41009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.1:45881
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.229.1
--webserver_port=36969
--tserver_master_addrs=127.11.229.62:34709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.1
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:00.548496 28174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:00.549674 28174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:00.561184 28200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:00.561378 28174 server_base.cc:1034] running on GCE node
W20260923 17:28:00.561213 28199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:00.561204 28202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:00.562032 28174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:00.562795 28174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:00.563957 28174 hybrid_clock.cc:648] HybridClock initialized: now 1790184480563925 us; error 49 us; skew 500 ppm
Sep 23 17:28:00 dist-test-slave-ptl9 krb5kdc[27555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184480, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:00.566959 28174 init.cc:377] Logged in from keytab as kudu/127.11.229.1@KRBTEST.COM (short username kudu)
I20260923 17:28:00.567816 28174 webserver.cc:460] Webserver started at http://127.11.229.1:36969/ using document root <none> and password file <none>
I20260923 17:28:00.568408 28174 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:00.568539 28174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:00.572752 28174 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260923 17:28:00.575590 28209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:00.576975 28174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260923 17:28:00.577121 28174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/wal
uuid: "cd15e54eada447f5b9e949c46b37a1b0"
format_stamp: "Formatted at 2026-09-23 17:27:39 on dist-test-slave-ptl9"
server_key: "1d28c559d8bb506051c0f17f650cb8bd"
server_key_iv: "19035947c5e6b01ded0a1325a0ac2a2f"
server_key_version: "encryptionkey@0"
I20260923 17:28:00.577765 28174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:00.591526 28174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:00.594899 28174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:00.595113 28174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:00.595736 28174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:00.596928 28216 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 17:28:00.602530 28174 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 17:28:00.602631 28174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260923 17:28:00.602732 28174 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 17:28:00.604550 28174 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 17:28:00.604626 28174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260923 17:28:00.605115 28216 tablet_bootstrap.cc:492] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0: Bootstrap starting.
I20260923 17:28:00.615784 28174 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.1:45881
I20260923 17:28:00.615864 28323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.1:45881 every 8 connection(s)
I20260923 17:28:00.616818 28174 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
I20260923 17:28:00.619961 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 28174
I20260923 17:28:00.620424 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.2:42517
--local_ip_for_outbound_sockets=127.11.229.2
--tserver_master_addrs=127.11.229.62:34709
--webserver_port=42393
--webserver_interface=127.11.229.2
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:41009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:28:00.626629 28216 log.cc:824] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0: Log is configured to *not* fsync() on all Append() calls
Sep 23 17:28:00 dist-test-slave-ptl9 krb5kdc[27555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184480, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:28:00.635107 28216 tablet_bootstrap.cc:492] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:28:00.635648 28216 tablet_bootstrap.cc:492] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0: Bootstrap complete.
I20260923 17:28:00.637068 28216 ts_tablet_manager.cc:1397] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0: Time spent bootstrapping tablet: real 0.032s user 0.026s sys 0.004s
I20260923 17:28:00.637080 28326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:00.619236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:52609 (local address 127.11.229.62:34709)
0923 17:28:00.619698 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:00.619728 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:00.621080 (+ 1352us) server_negotiation.cc:407] Connection header received
0923 17:28:00.622634 (+ 1554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:00.622645 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:00.622961 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:00.623303 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:00.624482 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:00.625614 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:00.626789 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:00.627404 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:00.627575 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:00.630869 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:00.630880 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:00.630902 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:00.630942 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:00.633353 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:00.633988 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:00.633993 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:00.633999 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:00.634149 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:00.634562 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:00.634566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:00.634567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:00.635026 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:00.635159 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:00.635898 (+ 739us) server_negotiation.cc:299] Negotiation successful
0923 17:28:00.636207 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":130,"threads_started":1}
I20260923 17:28:00.637154 28327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:00.619145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.1:52609)
0923 17:28:00.620847 (+ 1702us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:00.620915 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:00.622174 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:00.623376 (+ 1202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:00.623385 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:00.623816 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:00.624293 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:00.624308 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:00.625759 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:00.625768 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:00.626653 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:00.626663 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:00.627578 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:00.627584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:00.627860 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:00.628623 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:00.628651 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:00.630618 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:00.633498 (+ 2880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:00.633504 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:00.633516 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:00.633867 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:00.634280 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:00.634283 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:00.634285 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:00.634460 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:00.635160 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:00.635168 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:00.635482 (+ 314us) client_negotiation.cc:769] Sending connection context
0923 17:28:00.635713 (+ 231us) client_negotiation.cc:240] Negotiation successful
0923 17:28:00.636079 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":140,"threads_started":1}
I20260923 17:28:00.640139 28324 heartbeater.cc:347] Connected to a master server at 127.11.229.62:34709
I20260923 17:28:00.639897 28216 raft_consensus.cc:348] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d05fb754494f3582f4d96f1d3dd5f0" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34709 } } peers { permanent_uuid: "97d1965102a7423f87a0de343bdbd507" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42517 } } peers { permanent_uuid: "cd15e54eada447f5b9e949c46b37a1b0" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45881 } }
I20260923 17:28:00.640379 28324 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:00.640403 28216 raft_consensus.cc:729] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd15e54eada447f5b9e949c46b37a1b0, State: Initialized, Role: FOLLOWER
I20260923 17:28:00.640880 28216 consensus_queue.cc:260] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d05fb754494f3582f4d96f1d3dd5f0" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34709 } } peers { permanent_uuid: "97d1965102a7423f87a0de343bdbd507" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42517 } } peers { permanent_uuid: "cd15e54eada447f5b9e949c46b37a1b0" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45881 } }
I20260923 17:28:00.640957 28324 heartbeater.cc:511] Master 127.11.229.62:34709 requested a full tablet report, sending...
I20260923 17:28:00.641995 28216 ts_tablet_manager.cc:1428] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260923 17:28:00.643920 28136 ts_manager.cc:194] Registered new tserver with Master: cd15e54eada447f5b9e949c46b37a1b0 (127.11.229.1:45881)
I20260923 17:28:00.648725 28136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.1@KRBTEST.COM'} at 127.11.229.1:52609
I20260923 17:28:00.651598 28324 heartbeater.cc:503] Master 127.11.229.62:34709 was elected leader, sending a full tablet report...
Sep 23 17:28:00 dist-test-slave-ptl9 krb5kdc[27555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184480, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for kudu/127.11.229.1@KRBTEST.COM
I20260923 17:28:00.658500 28331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:00.647438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:45881 (local address 127.0.0.1:45436)
0923 17:28:00.648078 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:00.648126 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:00.648316 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:00.648808 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:00.648814 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:00.649227 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:00.649429 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:00.649455 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:00.650777 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:00.650784 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:00.651595 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:00.651606 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:00.652252 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:00.652256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:00.652371 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:00.652979 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:00.652991 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:00.654554 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:00.656531 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:00.656542 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:00.656548 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:00.656796 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:00.657108 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:00.657112 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:00.657114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:00.657205 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:00.657818 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:00.657824 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:00.657956 (+ 132us) client_negotiation.cc:769] Sending connection context
0923 17:28:00.658093 (+ 137us) client_negotiation.cc:240] Negotiation successful
0923 17:28:00.658305 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":120,"threads_started":1}
I20260923 17:28:00.658524 28332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:00.647870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45436 (local address 127.11.229.1:45881)
0923 17:28:00.648194 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:00.648208 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:00.648233 (+ 25us) server_negotiation.cc:407] Connection header received
0923 17:28:00.648481 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:00.648488 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:00.648649 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:00.648805 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:00.649636 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:00.650663 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:00.651790 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:00.652122 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:00.652248 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:00.654702 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:00.654711 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:00.654724 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:00.654749 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:00.656409 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:00.656917 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:00.656922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:00.656924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:00.657009 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:00.657319 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:00.657322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:00.657324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:00.657700 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:00.657846 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:00.658216 (+ 370us) server_negotiation.cc:299] Negotiation successful
0923 17:28:00.658381 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":109,"threads_started":1}
I20260923 17:28:00.660135 28258 tablet_service.cc:1514] Processing DeleteTablet for tablet d61349c609ee440fa0ff7eebcdecbc18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:27:40 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:45436
I20260923 17:28:00.661007 28333 tablet_replica.cc:330] stopping tablet replica
I20260923 17:28:00.661382 28333 raft_consensus.cc:2229] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:28:00.661662 28333 raft_consensus.cc:2258] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:00.662686 28333 ts_tablet_manager.cc:1905] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:00.666551 28333 ts_tablet_manager.cc:1918] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:28:00.666635 28333 log.cc:1196] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/wal/wals/d61349c609ee440fa0ff7eebcdecbc18
I20260923 17:28:00.666904 28333 ts_tablet_manager.cc:1939] T d61349c609ee440fa0ff7eebcdecbc18 P cd15e54eada447f5b9e949c46b37a1b0: Deleting consensus metadata
I20260923 17:28:00.668864 28122 catalog_manager.cc:4841] TS cd15e54eada447f5b9e949c46b37a1b0 (127.11.229.1:45881): tablet d61349c609ee440fa0ff7eebcdecbc18 (table test-table [id=38eb2a456b864c2da48a0aa3ec972f55]) successfully deleted
W20260923 17:28:00.743119 28328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:00.743412 28328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:00.743503 28328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:00.747557 28328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:00.747642 28328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:00.747727 28328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.2
I20260923 17:28:00.753137 28328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:41009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.2:42517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.229.2
--webserver_port=42393
--tserver_master_addrs=127.11.229.62:34709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.2
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:00.754259 28328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:00.755306 28328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:00.764994 28341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:00.764994 28338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:00.765015 28339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:00.765064 28328 server_base.cc:1034] running on GCE node
I20260923 17:28:00.765672 28328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:00.766314 28328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:00.767499 28328 hybrid_clock.cc:648] HybridClock initialized: now 1790184480767492 us; error 30 us; skew 500 ppm
Sep 23 17:28:00 dist-test-slave-ptl9 krb5kdc[27555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184480, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:00.770301 28328 init.cc:377] Logged in from keytab as kudu/127.11.229.2@KRBTEST.COM (short username kudu)
I20260923 17:28:00.771153 28328 webserver.cc:460] Webserver started at http://127.11.229.2:42393/ using document root <none> and password file <none>
I20260923 17:28:00.771731 28328 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:00.771835 28328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:00.775766 28328 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:28:00.777920 28348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:00.779143 28328 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:28:00.779280 28328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/wal
uuid: "97d1965102a7423f87a0de343bdbd507"
format_stamp: "Formatted at 2026-09-23 17:27:39 on dist-test-slave-ptl9"
server_key: "46a87e6b4f1a26a1f002e65875cd1473"
server_key_iv: "d6a284028d11b851ddcc93c3be748ccb"
server_key_version: "encryptionkey@0"
I20260923 17:28:00.779855 28328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:00.795060 28328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:00.798615 28328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:00.798831 28328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:00.799465 28328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:00.800338 28328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:00.800411 28328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:00.800485 28328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:00.800539 28328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:00.809856 28328 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.2:42517
I20260923 17:28:00.809875 28461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.2:42517 every 8 connection(s)
I20260923 17:28:00.810922 28328 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
I20260923 17:28:00.818231 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 28328
I20260923 17:28:00.818768 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.3:34709
--local_ip_for_outbound_sockets=127.11.229.3
--tserver_master_addrs=127.11.229.62:34709
--webserver_port=33533
--webserver_interface=127.11.229.3
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:41009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:28:00 dist-test-slave-ptl9 krb5kdc[27555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184480, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:28:00.824963 28326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:00.812879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:57039 (local address 127.11.229.62:34709)
0923 17:28:00.813062 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:00.813067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:00.814211 (+ 1144us) server_negotiation.cc:407] Connection header received
0923 17:28:00.815083 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:00.815086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:00.815139 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:00.815215 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:00.816402 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:00.816935 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:00.818125 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:00.818480 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:00.818606 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:00.821233 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:00.821237 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:00.821239 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:00.821308 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:00.822797 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:00.823352 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:00.823362 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:00.823364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:00.823445 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:00.823802 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:00.823806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:00.823808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:00.824010 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:00.824119 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:00.824643 (+ 524us) server_negotiation.cc:299] Negotiation successful
0923 17:28:00.824791 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 17:28:00.825932 28464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:00.813391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.2:57039)
0923 17:28:00.814058 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:00.814100 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:00.814897 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:00.815403 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:00.815410 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:00.815769 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:00.816271 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:00.816286 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:00.817053 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:00.817056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:00.818000 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:00.818011 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:00.818639 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:00.818642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:00.818788 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:00.819487 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:00.819503 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:00.821038 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:00.822928 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:00.822935 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:00.822954 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:00.823231 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:00.823545 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:00.823548 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:00.823550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:00.823702 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:00.824119 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:00.824127 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:00.824397 (+ 270us) client_negotiation.cc:769] Sending connection context
0923 17:28:00.824603 (+ 206us) client_negotiation.cc:240] Negotiation successful
0923 17:28:00.824973 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":111,"threads_started":1}
I20260923 17:28:00.827452 28462 heartbeater.cc:347] Connected to a master server at 127.11.229.62:34709
I20260923 17:28:00.827716 28462 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:00.828244 28462 heartbeater.cc:511] Master 127.11.229.62:34709 requested a full tablet report, sending...
I20260923 17:28:00.829480 28136 ts_manager.cc:194] Registered new tserver with Master: 97d1965102a7423f87a0de343bdbd507 (127.11.229.2:42517)
I20260923 17:28:00.830289 28136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.2@KRBTEST.COM'} at 127.11.229.2:57039
W20260923 17:28:00.933534 28465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:00.933823 28465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:00.933871 28465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:00.937803 28465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:00.937860 28465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:00.937918 28465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.3
I20260923 17:28:00.943333 28465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:41009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.3:34709
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.229.3
--webserver_port=33533
--tserver_master_addrs=127.11.229.62:34709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.3
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:00.944375 28465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:00.945441 28465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:00.955407 28470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:00.955394 28473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:00.955603 28465 server_base.cc:1034] running on GCE node
W20260923 17:28:00.955660 28471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:00.956141 28465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:00.956853 28465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:00.958038 28465 hybrid_clock.cc:648] HybridClock initialized: now 1790184480958021 us; error 30 us; skew 500 ppm
Sep 23 17:28:00 dist-test-slave-ptl9 krb5kdc[27555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184480, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:00.960876 28465 init.cc:377] Logged in from keytab as kudu/127.11.229.3@KRBTEST.COM (short username kudu)
I20260923 17:28:00.961854 28465 webserver.cc:460] Webserver started at http://127.11.229.3:33533/ using document root <none> and password file <none>
I20260923 17:28:00.962427 28465 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:00.962538 28465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:00.966701 28465 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:28:00.968760 28480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:00.970090 28465 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:28:00.970232 28465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/wal
uuid: "76d05fb754494f3582f4d96f1d3dd5f0"
format_stamp: "Formatted at 2026-09-23 17:27:39 on dist-test-slave-ptl9"
server_key: "986939a2dd0d35b6bff2290816021f51"
server_key_iv: "26c775f349b525de743936e442fc1b76"
server_key_version: "encryptionkey@0"
I20260923 17:28:00.970800 28465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:00.985451 28465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:00.988776 28465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:00.988991 28465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:00.989660 28465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:00.990513 28465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:00.990584 28465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:00.990658 28465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:00.990702 28465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:00.999615 28465 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.3:34709
I20260923 17:28:00.999653 28593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.3:34709 every 8 connection(s)
I20260923 17:28:01.000639 28465 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
I20260923 17:28:01.005705 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 28465
Sep 23 17:28:01 dist-test-slave-ptl9 krb5kdc[27555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184480, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:28:01.014585 28598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:01.006419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47258 (local address 127.11.229.62:34709)
0923 17:28:01.006749 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:01.006756 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:01.006770 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:28:01.006936 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:01.006942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:01.006991 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:01.007084 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:01.007800 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.008330 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.009225 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.009553 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.009665 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:01.011108 (+ 1443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:01.011111 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:01.011113 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:01.011135 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:01.012765 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:01.013192 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:01.013196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:01.013198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:01.013327 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:01.013610 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:01.013614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:01.013615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:01.013888 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:01.014105 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:01.014261 (+ 156us) server_negotiation.cc:299] Negotiation successful
0923 17:28:01.014393 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":75,"threads_started":1}
I20260923 17:28:01.015919 28326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:01.002515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:53601 (local address 127.11.229.62:34709)
0923 17:28:01.002661 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:01.002665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:01.003737 (+ 1072us) server_negotiation.cc:407] Connection header received
0923 17:28:01.004704 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:01.004707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:01.004769 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:01.004881 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:01.006959 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.007503 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.008654 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.008886 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.009004 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:01.012031 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:01.012037 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:01.012042 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:01.012086 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:01.013640 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:01.014227 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:01.014230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:01.014232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:01.014315 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:01.014705 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:01.014710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:01.014713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:01.014958 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:01.015069 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:01.015660 (+ 591us) server_negotiation.cc:299] Negotiation successful
0923 17:28:01.015769 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 17:28:01.016932 28596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:01.002919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.3:53601)
0923 17:28:01.003581 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:01.003622 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:01.004481 (+ 859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:01.005066 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:01.005073 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:01.006197 (+ 1124us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:01.006809 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:01.006824 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.007629 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.007633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:01.008524 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:01.008534 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.009045 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.009049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:01.009253 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:01.009964 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:01.009988 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:01.011865 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:01.013749 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:01.013754 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:01.013776 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:01.014121 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:01.014419 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:01.014422 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:01.014424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:01.014593 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:01.015070 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:01.015079 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:01.015416 (+ 337us) client_negotiation.cc:769] Sending connection context
0923 17:28:01.015622 (+ 206us) client_negotiation.cc:240] Negotiation successful
0923 17:28:01.015967 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":110,"threads_started":1}
I20260923 17:28:01.018093 28594 heartbeater.cc:347] Connected to a master server at 127.11.229.62:34709
I20260923 17:28:01.018352 28594 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:01.018836 28594 heartbeater.cc:511] Master 127.11.229.62:34709 requested a full tablet report, sending...
I20260923 17:28:01.019807 28136 ts_manager.cc:194] Registered new tserver with Master: 76d05fb754494f3582f4d96f1d3dd5f0 (127.11.229.3:34709)
I20260923 17:28:01.020366 28136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.3@KRBTEST.COM'} at 127.11.229.3:53601
I20260923 17:28:01.027904 12180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:28:01.031594 28326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:01.028509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47274 (local address 127.11.229.62:34709)
0923 17:28:01.028623 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:01.028626 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:01.028778 (+ 152us) server_negotiation.cc:407] Connection header received
0923 17:28:01.028928 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:01.028930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:01.029020 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:01.029101 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:28:01.029255 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.029809 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.030615 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.030830 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.030905 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:01.031164 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:28:01.031296 (+ 132us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0923 17:28:01.031497 (+ 201us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47274: token expired
Metrics: {"server-negotiator.queue_time_us":45}
W20260923 17:28:01.031679 28326 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47274: token expired
W20260923 17:28:01.032022 28063 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.11.229.62:34709)
I20260923 17:28:01.039352 28326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:01.032385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47276 (local address 127.11.229.62:34709)
0923 17:28:01.032498 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:01.032503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:01.032565 (+ 62us) server_negotiation.cc:407] Connection header received
0923 17:28:01.032695 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:01.032698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:01.032748 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:01.032830 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:01.033525 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.034033 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.034847 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.035061 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.035170 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:01.036476 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:01.036480 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:01.036483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:01.036515 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:01.037712 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:01.038156 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:01.038165 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:01.038166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:01.038242 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:01.038515 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:01.038522 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:01.038523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:01.038692 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:01.038839 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:01.039078 (+ 239us) server_negotiation.cc:299] Negotiation successful
0923 17:28:01.039191 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 17:28:01.041772 28136 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:28:01.045104 28136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:28:01.052826 28258 tablet_service.cc:1467] Processing CreateTablet for tablet 5ad77abf6b7c4bebbb7978f0dbdf240a (DEFAULT_TABLE table=test-table [id=b9cc8a72935140a28469247aec9333d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:01.053521 28258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ad77abf6b7c4bebbb7978f0dbdf240a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:01.056744 28216 tablet_bootstrap.cc:492] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0: Bootstrap starting.
I20260923 17:28:01.057710 28216 tablet_bootstrap.cc:654] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:01.058513 28331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:01.051810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:42517 (local address 127.0.0.1:46976)
0923 17:28:01.051907 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:01.051923 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:01.052001 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:01.052978 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:01.052981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:01.053002 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:01.053110 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:01.053129 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.054650 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.054654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:01.055873 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:01.055884 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.057934 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.057938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:01.058098 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:01.058145 (+ 47us) client_negotiation.cc:769] Sending connection context
0923 17:28:01.058266 (+ 121us) client_negotiation.cc:240] Negotiation successful
0923 17:28:01.058357 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":24}
I20260923 17:28:01.058725 28603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:01.052020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46976 (local address 127.11.229.2:42517)
0923 17:28:01.052411 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:01.052426 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:01.052453 (+ 27us) server_negotiation.cc:407] Connection header received
0923 17:28:01.052540 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:01.052546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:01.052762 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:01.052977 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:01.053295 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.054519 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.056142 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.057824 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.057934 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:01.058104 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:01.058444 (+ 340us) server_negotiation.cc:299] Negotiation successful
0923 17:28:01.058564 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":170,"threads_started":1}
I20260923 17:28:01.059636 28216 tablet_bootstrap.cc:492] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0: No bootstrap required, opened a new log
I20260923 17:28:01.059722 28216 ts_tablet_manager.cc:1397] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 17:28:01.060155 28216 raft_consensus.cc:348] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d1965102a7423f87a0de343bdbd507" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42517 } } peers { permanent_uuid: "76d05fb754494f3582f4d96f1d3dd5f0" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34709 } } peers { permanent_uuid: "cd15e54eada447f5b9e949c46b37a1b0" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45881 } }
I20260923 17:28:01.060280 28216 raft_consensus.cc:374] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:01.060328 28216 raft_consensus.cc:729] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd15e54eada447f5b9e949c46b37a1b0, State: Initialized, Role: FOLLOWER
I20260923 17:28:01.060459 28216 consensus_queue.cc:260] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d1965102a7423f87a0de343bdbd507" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42517 } } peers { permanent_uuid: "76d05fb754494f3582f4d96f1d3dd5f0" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34709 } } peers { permanent_uuid: "cd15e54eada447f5b9e949c46b37a1b0" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45881 } }
I20260923 17:28:01.060847 28216 ts_tablet_manager.cc:1428] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 17:28:01.062270 28396 tablet_service.cc:1467] Processing CreateTablet for tablet 5ad77abf6b7c4bebbb7978f0dbdf240a (DEFAULT_TABLE table=test-table [id=b9cc8a72935140a28469247aec9333d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:01.063035 28604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:01.052111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:34709 (local address 127.0.0.1:54250)
0923 17:28:01.056975 (+ 4864us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:01.056990 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:01.057091 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:01.057861 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:01.057866 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:01.057891 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:01.058043 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:01.058052 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.059940 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.059943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:01.061128 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:01.061140 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.062514 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.062518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:01.062630 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:01.062656 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 17:28:01.062759 (+ 103us) client_negotiation.cc:240] Negotiation successful
0923 17:28:01.062903 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4776,"mutex_wait_us":28,"thread_start_us":115,"threads_started":1}
I20260923 17:28:01.063383 28605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:01.052555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54250 (local address 127.11.229.3:34709)
0923 17:28:01.052939 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:01.052952 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:01.057104 (+ 4152us) server_negotiation.cc:407] Connection header received
0923 17:28:01.057202 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:01.057209 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:01.057464 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:01.057637 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:01.058722 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.059779 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.061303 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.062175 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.062280 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:01.062450 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:01.062792 (+ 342us) server_negotiation.cc:299] Negotiation successful
0923 17:28:01.062894 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":146,"threads_started":1}
I20260923 17:28:01.063481 28396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ad77abf6b7c4bebbb7978f0dbdf240a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:01.066375 28528 tablet_service.cc:1467] Processing CreateTablet for tablet 5ad77abf6b7c4bebbb7978f0dbdf240a (DEFAULT_TABLE table=test-table [id=b9cc8a72935140a28469247aec9333d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:01.067473 28528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ad77abf6b7c4bebbb7978f0dbdf240a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:01.072057 28607 tablet_bootstrap.cc:492] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 97d1965102a7423f87a0de343bdbd507: Bootstrap starting.
I20260923 17:28:01.074590 28607 tablet_bootstrap.cc:654] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 97d1965102a7423f87a0de343bdbd507: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:01.075441 28608 tablet_bootstrap.cc:492] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0: Bootstrap starting.
I20260923 17:28:01.075462 28607 log.cc:824] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 97d1965102a7423f87a0de343bdbd507: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:01.077961 28607 tablet_bootstrap.cc:492] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 97d1965102a7423f87a0de343bdbd507: No bootstrap required, opened a new log
I20260923 17:28:01.078203 28607 ts_tablet_manager.cc:1397] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 97d1965102a7423f87a0de343bdbd507: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260923 17:28:01.078215 28608 tablet_bootstrap.cc:654] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:01.079068 28608 log.cc:824] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:01.081146 28608 tablet_bootstrap.cc:492] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0: No bootstrap required, opened a new log
I20260923 17:28:01.081379 28608 ts_tablet_manager.cc:1397] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260923 17:28:01.082185 28607 raft_consensus.cc:348] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 97d1965102a7423f87a0de343bdbd507 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d1965102a7423f87a0de343bdbd507" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42517 } } peers { permanent_uuid: "76d05fb754494f3582f4d96f1d3dd5f0" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34709 } } peers { permanent_uuid: "cd15e54eada447f5b9e949c46b37a1b0" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45881 } }
I20260923 17:28:01.082481 28607 raft_consensus.cc:374] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 97d1965102a7423f87a0de343bdbd507 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:01.082540 28607 raft_consensus.cc:729] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 97d1965102a7423f87a0de343bdbd507 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97d1965102a7423f87a0de343bdbd507, State: Initialized, Role: FOLLOWER
I20260923 17:28:01.083041 28607 consensus_queue.cc:260] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 97d1965102a7423f87a0de343bdbd507 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d1965102a7423f87a0de343bdbd507" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42517 } } peers { permanent_uuid: "76d05fb754494f3582f4d96f1d3dd5f0" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34709 } } peers { permanent_uuid: "cd15e54eada447f5b9e949c46b37a1b0" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45881 } }
I20260923 17:28:01.083729 28462 heartbeater.cc:503] Master 127.11.229.62:34709 was elected leader, sending a full tablet report...
I20260923 17:28:01.084115 28607 ts_tablet_manager.cc:1428] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 97d1965102a7423f87a0de343bdbd507: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 17:28:01.085088 28608 raft_consensus.cc:348] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d1965102a7423f87a0de343bdbd507" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42517 } } peers { permanent_uuid: "76d05fb754494f3582f4d96f1d3dd5f0" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34709 } } peers { permanent_uuid: "cd15e54eada447f5b9e949c46b37a1b0" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45881 } }
I20260923 17:28:01.085361 28608 raft_consensus.cc:374] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:01.085430 28608 raft_consensus.cc:729] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d05fb754494f3582f4d96f1d3dd5f0, State: Initialized, Role: FOLLOWER
I20260923 17:28:01.085968 28608 consensus_queue.cc:260] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d1965102a7423f87a0de343bdbd507" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42517 } } peers { permanent_uuid: "76d05fb754494f3582f4d96f1d3dd5f0" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34709 } } peers { permanent_uuid: "cd15e54eada447f5b9e949c46b37a1b0" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45881 } }
I20260923 17:28:01.086594 28594 heartbeater.cc:503] Master 127.11.229.62:34709 was elected leader, sending a full tablet report...
I20260923 17:28:01.086966 28608 ts_tablet_manager.cc:1428] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260923 17:28:01.118036 28325 tablet.cc:2369] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:28:01.251677 28595 tablet.cc:2369] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:28:01.312405 28463 tablet.cc:2369] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 97d1965102a7423f87a0de343bdbd507: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:28:01.408100 28167 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.11.229.62:34709
I20260923 17:28:01.415040 28612 raft_consensus.cc:482] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:28:01.415257 28612 raft_consensus.cc:504] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d1965102a7423f87a0de343bdbd507" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42517 } } peers { permanent_uuid: "76d05fb754494f3582f4d96f1d3dd5f0" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34709 } } peers { permanent_uuid: "cd15e54eada447f5b9e949c46b37a1b0" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45881 } }
I20260923 17:28:01.416564 28612 leader_election.cc:288] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97d1965102a7423f87a0de343bdbd507 (127.11.229.2:42517), cd15e54eada447f5b9e949c46b37a1b0 (127.11.229.1:45881)
I20260923 17:28:01.421484 28613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:01.416972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:40239 (local address 127.11.229.1:45881)
0923 17:28:01.417359 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:01.417363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:01.417559 (+ 196us) server_negotiation.cc:407] Connection header received
0923 17:28:01.417626 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:01.417628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:01.417682 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:01.417814 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:01.418057 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.418788 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.420191 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.420857 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.420966 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:01.421116 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:01.421254 (+ 138us) server_negotiation.cc:299] Negotiation successful
0923 17:28:01.421366 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":94,"threads_started":1}
I20260923 17:28:01.421480 28603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:01.416992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:45693 (local address 127.11.229.2:42517)
0923 17:28:01.417159 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:01.417164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:01.417181 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:28:01.417245 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:01.417250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:01.417353 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:01.417435 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:01.417759 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.418557 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.420191 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.420767 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.420913 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:01.420981 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:01.421254 (+ 273us) server_negotiation.cc:299] Negotiation successful
0923 17:28:01.421348 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 17:28:01.421509 28596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:01.416775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:42517 (local address 127.11.229.3:45693)
0923 17:28:01.416959 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:01.416975 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:01.417076 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:01.417473 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:01.417477 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:01.417501 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:01.417635 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:01.417648 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.418709 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.418712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:01.420054 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:01.420066 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.420929 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.420934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:01.421081 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:01.421137 (+ 56us) client_negotiation.cc:769] Sending connection context
0923 17:28:01.421219 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 17:28:01.421351 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":47}
I20260923 17:28:01.421517 28614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:01.416775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:45881 (local address 127.11.229.3:40239)
0923 17:28:01.417454 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:01.417467 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:01.417555 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:01.417841 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:01.417843 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:01.417858 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:01.417947 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:01.417952 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.418896 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.418899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:01.420074 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:01.420081 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.420961 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.420965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:01.421122 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:01.421141 (+ 19us) client_negotiation.cc:769] Sending connection context
0923 17:28:01.421219 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 17:28:01.421346 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":439,"thread_start_us":92,"threads_started":1}
I20260923 17:28:01.422260 28416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad77abf6b7c4bebbb7978f0dbdf240a" candidate_uuid: "76d05fb754494f3582f4d96f1d3dd5f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97d1965102a7423f87a0de343bdbd507" is_pre_election: true
I20260923 17:28:01.422264 28278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad77abf6b7c4bebbb7978f0dbdf240a" candidate_uuid: "76d05fb754494f3582f4d96f1d3dd5f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd15e54eada447f5b9e949c46b37a1b0" is_pre_election: true
I20260923 17:28:01.422561 28416 raft_consensus.cc:2454] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 97d1965102a7423f87a0de343bdbd507 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d05fb754494f3582f4d96f1d3dd5f0 in term 0.
I20260923 17:28:01.422561 28278 raft_consensus.cc:2454] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d05fb754494f3582f4d96f1d3dd5f0 in term 0.
I20260923 17:28:01.423110 28483 leader_election.cc:302] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 76d05fb754494f3582f4d96f1d3dd5f0, 97d1965102a7423f87a0de343bdbd507, cd15e54eada447f5b9e949c46b37a1b0; no voters:
I20260923 17:28:01.423413 28612 raft_consensus.cc:2781] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:28:01.423501 28612 raft_consensus.cc:482] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:28:01.423543 28612 raft_consensus.cc:3037] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:01.424569 28612 raft_consensus.cc:504] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d1965102a7423f87a0de343bdbd507" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42517 } } peers { permanent_uuid: "76d05fb754494f3582f4d96f1d3dd5f0" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34709 } } peers { permanent_uuid: "cd15e54eada447f5b9e949c46b37a1b0" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45881 } }
I20260923 17:28:01.424914 28612 leader_election.cc:288] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0 [CANDIDATE]: Term 1 election: Requested vote from peers 97d1965102a7423f87a0de343bdbd507 (127.11.229.2:42517), cd15e54eada447f5b9e949c46b37a1b0 (127.11.229.1:45881)
I20260923 17:28:01.425315 28416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad77abf6b7c4bebbb7978f0dbdf240a" candidate_uuid: "76d05fb754494f3582f4d96f1d3dd5f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97d1965102a7423f87a0de343bdbd507"
I20260923 17:28:01.425285 28278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad77abf6b7c4bebbb7978f0dbdf240a" candidate_uuid: "76d05fb754494f3582f4d96f1d3dd5f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd15e54eada447f5b9e949c46b37a1b0"
I20260923 17:28:01.425439 28278 raft_consensus.cc:3037] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:01.425441 28416 raft_consensus.cc:3037] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 97d1965102a7423f87a0de343bdbd507 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:01.426651 28278 raft_consensus.cc:2454] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d05fb754494f3582f4d96f1d3dd5f0 in term 1.
I20260923 17:28:01.426651 28416 raft_consensus.cc:2454] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 97d1965102a7423f87a0de343bdbd507 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d05fb754494f3582f4d96f1d3dd5f0 in term 1.
I20260923 17:28:01.426995 28483 leader_election.cc:302] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d05fb754494f3582f4d96f1d3dd5f0, cd15e54eada447f5b9e949c46b37a1b0; no voters:
I20260923 17:28:01.427201 28612 raft_consensus.cc:2781] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:01.427450 28612 raft_consensus.cc:686] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0 [term 1 LEADER]: Becoming Leader. State: Replica: 76d05fb754494f3582f4d96f1d3dd5f0, State: Running, Role: LEADER
I20260923 17:28:01.427790 28612 consensus_queue.cc:237] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d1965102a7423f87a0de343bdbd507" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42517 } } peers { permanent_uuid: "76d05fb754494f3582f4d96f1d3dd5f0" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34709 } } peers { permanent_uuid: "cd15e54eada447f5b9e949c46b37a1b0" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45881 } }
I20260923 17:28:01.431206 28136 catalog_manager.cc:5495] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76d05fb754494f3582f4d96f1d3dd5f0 (127.11.229.3). New cstate: current_term: 1 leader_uuid: "76d05fb754494f3582f4d96f1d3dd5f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d1965102a7423f87a0de343bdbd507" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76d05fb754494f3582f4d96f1d3dd5f0" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd15e54eada447f5b9e949c46b37a1b0" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45881 } health_report { overall_health: UNKNOWN } } }
I20260923 17:28:01.512283 28605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:01.508898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54252 (local address 127.11.229.3:34709)
0923 17:28:01.509035 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:01.509038 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:01.509052 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:28:01.509172 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:01.509174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:01.509229 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:01.509337 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:28:01.509565 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.510070 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.510913 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.511121 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.511195 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:01.511469 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:28:01.511892 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:28:01.511983 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:01.512039 (+ 56us) server_negotiation.cc:299] Negotiation successful
0923 17:28:01.512090 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 17:28:01.518553 28278 raft_consensus.cc:1261] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0 [term 1 FOLLOWER]: Refusing update from remote peer 76d05fb754494f3582f4d96f1d3dd5f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:28:01.518558 28416 raft_consensus.cc:1261] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 97d1965102a7423f87a0de343bdbd507 [term 1 FOLLOWER]: Refusing update from remote peer 76d05fb754494f3582f4d96f1d3dd5f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:28:01.519265 28615 consensus_queue.cc:1035] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd15e54eada447f5b9e949c46b37a1b0" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:28:01.519460 28612 consensus_queue.cc:1035] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97d1965102a7423f87a0de343bdbd507" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:28:01.526155 28620 mvcc.cc:205] Tried to move back new op lower bound from 7332595636292390912 to 7332595635930984448. Current Snapshot: MvccSnapshot[applied={T|T < 7332595636292390912}]
I20260923 17:28:01.526775 28619 mvcc.cc:205] Tried to move back new op lower bound from 7332595636292390912 to 7332595635930984448. Current Snapshot: MvccSnapshot[applied={T|T < 7332595636292390912}]
I20260923 17:28:01.527397 28618 mvcc.cc:205] Tried to move back new op lower bound from 7332595636292390912 to 7332595635930984448. Current Snapshot: MvccSnapshot[applied={T|T < 7332595636292390912}]
I20260923 17:28:01.535252 28135 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:28:01.535415 28135 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:28:01.537778 28135 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e: Sending DeleteTablet for 3 replicas of tablet 5ad77abf6b7c4bebbb7978f0dbdf240a
I20260923 17:28:01.538472 28396 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ad77abf6b7c4bebbb7978f0dbdf240a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:01 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:46976
I20260923 17:28:01.538514 28528 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ad77abf6b7c4bebbb7978f0dbdf240a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:01 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:54250
I20260923 17:28:01.538692 28258 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ad77abf6b7c4bebbb7978f0dbdf240a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:01 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:45436
I20260923 17:28:01.538996 28629 tablet_replica.cc:330] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 97d1965102a7423f87a0de343bdbd507: stopping tablet replica
I20260923 17:28:01.539023 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 28174
I20260923 17:28:01.539085 28628 tablet_replica.cc:330] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0: stopping tablet replica
I20260923 17:28:01.539372 28628 raft_consensus.cc:2229] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:28:01.539980 28628 raft_consensus.cc:2258] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:01.540014 28629 raft_consensus.cc:2229] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 97d1965102a7423f87a0de343bdbd507 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:28:01.540454 28629 raft_consensus.cc:2258] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 97d1965102a7423f87a0de343bdbd507 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:01.541782 28628 ts_tablet_manager.cc:1905] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:01.545450 28628 ts_tablet_manager.cc:1918] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:28:01.545619 28628 log.cc:1196] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/wal/wals/5ad77abf6b7c4bebbb7978f0dbdf240a
I20260923 17:28:01.546043 28628 ts_tablet_manager.cc:1939] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0: Deleting consensus metadata
I20260923 17:28:01.547262 28629 ts_tablet_manager.cc:1905] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 97d1965102a7423f87a0de343bdbd507: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260923 17:28:01.547387 28122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.229.1:45881 (error 108)
I20260923 17:28:01.547416 28120 catalog_manager.cc:4841] TS 76d05fb754494f3582f4d96f1d3dd5f0 (127.11.229.3:34709): tablet 5ad77abf6b7c4bebbb7978f0dbdf240a (table test-table [id=b9cc8a72935140a28469247aec9333d1]) successfully deleted
I20260923 17:28:01.548689 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 28328
W20260923 17:28:01.548735 28604 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:28:01.548165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:45881 (local address 127.0.0.1:45446)
0923 17:28:01.548563 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:01.548644 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.1:45881: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":297}
W20260923 17:28:01.550664 28122 catalog_manager.cc:4584] TS cd15e54eada447f5b9e949c46b37a1b0 (127.11.229.1:45881): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5ad77abf6b7c4bebbb7978f0dbdf240a: Network error: Client connection negotiation failed: client connection to 127.11.229.1:45881: connect: Connection reset by peer (error 104)
I20260923 17:28:01.555269 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 28465
I20260923 17:28:01.556095 28604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:01.555514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:42517 (local address 127.0.0.1:46990)
0923 17:28:01.555926 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:01.555990 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.2:42517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309}
I20260923 17:28:01.561587 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 28103
I20260923 17:28:21.568583 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:34709
--webserver_interface=127.11.229.62
--webserver_port=38351
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:41009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:34709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:21.682183 28632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:21.682520 28632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:21.682596 28632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:21.687234 28632 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 17:28:21.687289 28632 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 17:28:21.687306 28632 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 17:28:21.687320 28632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:28:21.687340 28632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:21.687357 28632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:21.687373 28632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:28:21.687389 28632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:28:21.692781 28632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:41009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:34709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:34709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=38351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:21.693964 28632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:21.694984 28632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:21.703931 28640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:21.704031 28632 server_base.cc:1034] running on GCE node
W20260923 17:28:21.703926 28637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:21.703931 28638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:21.704933 28632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:21.706247 28632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:21.707463 28632 hybrid_clock.cc:648] HybridClock initialized: now 1790184501707427 us; error 48 us; skew 500 ppm
Sep 23 17:28:21 dist-test-slave-ptl9 krb5kdc[27555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184501, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:21.710539 28632 init.cc:377] Logged in from keytab as kudu/127.11.229.62@KRBTEST.COM (short username kudu)
I20260923 17:28:21.711572 28632 webserver.cc:460] Webserver started at http://127.11.229.62:38351/ using document root <none> and password file <none>
I20260923 17:28:21.712110 28632 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:21.712266 28632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:21.716789 28632 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:28:21.719166 28647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:21.720633 28632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:28:21.720993 28632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "0aa575a5a84b41b194b099dc7e71f04e"
format_stamp: "Formatted at 2026-09-23 17:27:38 on dist-test-slave-ptl9"
server_key: "59c03647ad7833031233bee6ca0e32be"
server_key_iv: "a60248ed97d608168500c8060557ef64"
server_key_version: "encryptionkey@0"
I20260923 17:28:21.721699 28632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:21.733641 28632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:21.736802 28632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:21.737030 28632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:21.744375 28632 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:34709
I20260923 17:28:21.744656 28699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:34709 every 8 connection(s)
I20260923 17:28:21.745805 28632 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:28:21.750296 28700 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:28:21.752038 28700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e: Bootstrap starting.
I20260923 17:28:21.755669 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 28632
I20260923 17:28:21.756151 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.1:45881
--local_ip_for_outbound_sockets=127.11.229.1
--tserver_master_addrs=127.11.229.62:34709
--webserver_port=36969
--webserver_interface=127.11.229.1
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:41009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:28:21.771893 28700 log.cc:824] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:21.783679 28700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:28:21.784070 28700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e: Bootstrap complete.
I20260923 17:28:21.788167 28700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa575a5a84b41b194b099dc7e71f04e" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34709 } }
I20260923 17:28:21.788594 28700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aa575a5a84b41b194b099dc7e71f04e, State: Initialized, Role: FOLLOWER
I20260923 17:28:21.789098 28700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa575a5a84b41b194b099dc7e71f04e" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34709 } }
I20260923 17:28:21.789260 28700 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:28:21.789379 28700 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:28:21.789501 28700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [term 2 FOLLOWER]: Advancing to term 3
I20260923 17:28:21.791518 28700 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa575a5a84b41b194b099dc7e71f04e" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34709 } }
I20260923 17:28:21.791859 28700 leader_election.cc:302] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0aa575a5a84b41b194b099dc7e71f04e; no voters:
I20260923 17:28:21.792218 28700 leader_election.cc:288] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 17:28:21.792307 28704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [term 3 FOLLOWER]: Leader election won for term 3
I20260923 17:28:21.792574 28704 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [term 3 LEADER]: Becoming Leader. State: Replica: 0aa575a5a84b41b194b099dc7e71f04e, State: Running, Role: LEADER
I20260923 17:28:21.792936 28704 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa575a5a84b41b194b099dc7e71f04e" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34709 } }
I20260923 17:28:21.793552 28700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:28:21.794314 28705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0aa575a5a84b41b194b099dc7e71f04e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa575a5a84b41b194b099dc7e71f04e" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34709 } } }
I20260923 17:28:21.794453 28705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:21.795164 28706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0aa575a5a84b41b194b099dc7e71f04e. Latest consensus state: current_term: 3 leader_uuid: "0aa575a5a84b41b194b099dc7e71f04e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa575a5a84b41b194b099dc7e71f04e" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34709 } } }
I20260923 17:28:21.795264 28706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:21.796120 28711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:28:21.800189 28711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:28:21.800798 28711 catalog_manager.cc:1259] Loaded cluster ID: e7a40a4a7b144207bdc56630af64e984
I20260923 17:28:21.800911 28711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:28:21.802762 28711 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:28:21.803371 28711 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e: Loaded TSK: 0
I20260923 17:28:21.803457 28711 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0aa575a5a84b41b194b099dc7e71f04e: Loaded TSK: 1
F20260923 17:28:21.804060 28711 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f72d51519ad google::LogMessage::Fail() at ??:0
@ 0x7f72d515488a google::LogMessage::SendToLog() at ??:0
@ 0x7f72d51514de google::LogMessage::Flush() at ??:0
@ 0x7f72d5152a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f72d7e1518d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f72d7e0df0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f72dfab67bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f72dfa9099c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f72dfa91178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f72dfad2df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f72dfaf182b std::function<>::operator()() at ??:0
@ 0x7f72dfa90b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f72dfa91f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f72dfa8e579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f72dfad2640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f72d73a33d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f72d73a3c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f72d73a53f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f72d7394512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f72d89d86db start_thread at ??:0
@ 0x7f72d3c4f61f clone at ??:0
W20260923 17:28:21.911182 28702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:21.911442 28702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:21.911487 28702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:21.915627 28702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:21.915685 28702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:21.915774 28702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.1
I20260923 17:28:21.921083 28702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:41009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.1:45881
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.229.1
--webserver_port=36969
--tserver_master_addrs=127.11.229.62:34709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.1
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:21.922217 28702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:21.923272 28702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:21.933323 28748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:21.933380 28749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:21.933358 28751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:21.933653 28702 server_base.cc:1034] running on GCE node
I20260923 17:28:21.934303 28702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:21.935110 28702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:21.936321 28702 hybrid_clock.cc:648] HybridClock initialized: now 1790184501936295 us; error 36 us; skew 500 ppm
Sep 23 17:28:21 dist-test-slave-ptl9 krb5kdc[27555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184501, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:21.939996 28702 init.cc:377] Logged in from keytab as kudu/127.11.229.1@KRBTEST.COM (short username kudu)
I20260923 17:28:21.941195 28702 webserver.cc:460] Webserver started at http://127.11.229.1:36969/ using document root <none> and password file <none>
I20260923 17:28:21.941951 28702 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:21.942044 28702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:21.947396 28702 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260923 17:28:21.950284 28758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:21.951987 28702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:28:21.952127 28702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/wal
uuid: "cd15e54eada447f5b9e949c46b37a1b0"
format_stamp: "Formatted at 2026-09-23 17:27:39 on dist-test-slave-ptl9"
server_key: "1d28c559d8bb506051c0f17f650cb8bd"
server_key_iv: "19035947c5e6b01ded0a1325a0ac2a2f"
server_key_version: "encryptionkey@0"
I20260923 17:28:21.952874 28702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:21.983646 28702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:21.987811 28702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:21.988047 28702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:21.988726 28702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:21.990098 28765 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 17:28:21.996023 28702 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 17:28:21.996130 28702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260923 17:28:21.996248 28702 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 17:28:21.998172 28702 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 17:28:21.998262 28702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.001s
I20260923 17:28:21.998574 28765 tablet_bootstrap.cc:492] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0: Bootstrap starting.
I20260923 17:28:22.010496 28702 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.1:45881
I20260923 17:28:22.010679 28872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.1:45881 every 8 connection(s)
I20260923 17:28:22.011623 28702 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
I20260923 17:28:22.014397 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 28702
I20260923 17:28:22.014882 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.2:42517
--local_ip_for_outbound_sockets=127.11.229.2
--tserver_master_addrs=127.11.229.62:34709
--webserver_port=42393
--webserver_interface=127.11.229.2
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:41009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:28:22.021685 28765 log.cc:824] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:22.031374 28765 tablet_bootstrap.cc:492] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:28:22.031967 28765 tablet_bootstrap.cc:492] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0: Bootstrap complete.
I20260923 17:28:22.033485 28765 ts_tablet_manager.cc:1397] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0: Time spent bootstrapping tablet: real 0.035s user 0.024s sys 0.010s
I20260923 17:28:22.036857 28765 raft_consensus.cc:348] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d1965102a7423f87a0de343bdbd507" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42517 } } peers { permanent_uuid: "76d05fb754494f3582f4d96f1d3dd5f0" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34709 } } peers { permanent_uuid: "cd15e54eada447f5b9e949c46b37a1b0" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45881 } }
I20260923 17:28:22.037531 28765 raft_consensus.cc:729] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd15e54eada447f5b9e949c46b37a1b0, State: Initialized, Role: FOLLOWER
I20260923 17:28:22.038136 28765 consensus_queue.cc:260] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d1965102a7423f87a0de343bdbd507" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42517 } } peers { permanent_uuid: "76d05fb754494f3582f4d96f1d3dd5f0" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34709 } } peers { permanent_uuid: "cd15e54eada447f5b9e949c46b37a1b0" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45881 } }
I20260923 17:28:22.039319 28765 ts_tablet_manager.cc:1428] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
W20260923 17:28:22.052804 28875 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:28:22.013833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.1:56625)
0923 17:28:22.015202 (+ 1369us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:22.015266 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:22.016531 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:22.051930 (+ 35399us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1114,"thread_start_us":160,"threads_started":1}
W20260923 17:28:22.053431 28873 heartbeater.cc:644] Failed to heartbeat to 127.11.229.62:34709 (0 consecutive failures): Network error: Failed to ping master at 127.11.229.62:34709: Client connection negotiation failed: client connection to 127.11.229.62:34709: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 17:28:22.054229 28875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:22.053960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.1:42655)
0923 17:28:22.054093 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:22.054170 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260923 17:28:22.055025 28875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:22.054817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.1:47793)
0923 17:28:22.054930 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:22.054975 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260923 17:28:22.055227 28873 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 17:28:22.137036 28876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:22.137373 28876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:22.137454 28876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:22.141536 28876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:22.141593 28876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:22.141685 28876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.2
I20260923 17:28:22.148310 28876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:41009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.2:42517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.229.2
--webserver_port=42393
--tserver_master_addrs=127.11.229.62:34709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.2
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:22.149516 28876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:22.150583 28876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:22.161319 28883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:22.161434 28876 server_base.cc:1034] running on GCE node
W20260923 17:28:22.161363 28884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:22.161286 28886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:22.162034 28876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:22.162738 28876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:22.163981 28876 hybrid_clock.cc:648] HybridClock initialized: now 1790184502163948 us; error 54 us; skew 500 ppm
Sep 23 17:28:22 dist-test-slave-ptl9 krb5kdc[27555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184502, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:22.167021 28876 init.cc:377] Logged in from keytab as kudu/127.11.229.2@KRBTEST.COM (short username kudu)
I20260923 17:28:22.167974 28876 webserver.cc:460] Webserver started at http://127.11.229.2:42393/ using document root <none> and password file <none>
I20260923 17:28:22.168627 28876 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:22.168730 28876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:22.173067 28876 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:28:22.175693 28893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:22.177505 28876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:28:22.177663 28876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/wal
uuid: "97d1965102a7423f87a0de343bdbd507"
format_stamp: "Formatted at 2026-09-23 17:27:39 on dist-test-slave-ptl9"
server_key: "46a87e6b4f1a26a1f002e65875cd1473"
server_key_iv: "d6a284028d11b851ddcc93c3be748ccb"
server_key_version: "encryptionkey@0"
I20260923 17:28:22.178298 28876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:22.205456 28876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:22.208734 28876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:22.208969 28876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:22.209666 28876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:22.210824 28900 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 17:28:22.215153 28876 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 17:28:22.215245 28876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260923 17:28:22.215353 28876 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 17:28:22.217427 28876 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 17:28:22.217500 28876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260923 17:28:22.217779 28900 tablet_bootstrap.cc:492] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 97d1965102a7423f87a0de343bdbd507: Bootstrap starting.
I20260923 17:28:22.223572 28900 log.cc:824] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 97d1965102a7423f87a0de343bdbd507: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:22.228506 28876 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.2:42517
I20260923 17:28:22.228577 29008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.2:42517 every 8 connection(s)
I20260923 17:28:22.229485 28876 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
I20260923 17:28:22.231992 28900 tablet_bootstrap.cc:492] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 97d1965102a7423f87a0de343bdbd507: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:28:22.232404 28900 tablet_bootstrap.cc:492] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 97d1965102a7423f87a0de343bdbd507: Bootstrap complete.
I20260923 17:28:22.232748 28900 ts_tablet_manager.cc:1397] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 97d1965102a7423f87a0de343bdbd507: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20260923 17:28:22.232788 29011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:22.231377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.2:55491)
0923 17:28:22.231879 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:22.232007 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":160,"threads_started":1}
W20260923 17:28:22.233351 29009 heartbeater.cc:644] Failed to heartbeat to 127.11.229.62:34709 (0 consecutive failures): Network error: Failed to ping master at 127.11.229.62:34709: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
I20260923 17:28:22.233960 29011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:22.233735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.2:58173)
0923 17:28:22.233850 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:22.233907 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260923 17:28:22.234675 29011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:22.234479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.2:47679)
0923 17:28:22.234575 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:22.234616 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260923 17:28:22.235002 29009 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 17:28:22.235707 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 28876
I20260923 17:28:22.236176 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.3:34709
--local_ip_for_outbound_sockets=127.11.229.3
--tserver_master_addrs=127.11.229.62:34709
--webserver_port=33533
--webserver_interface=127.11.229.3
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:41009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:28:22.236133 28900 raft_consensus.cc:348] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 97d1965102a7423f87a0de343bdbd507 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d1965102a7423f87a0de343bdbd507" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42517 } } peers { permanent_uuid: "76d05fb754494f3582f4d96f1d3dd5f0" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34709 } } peers { permanent_uuid: "cd15e54eada447f5b9e949c46b37a1b0" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45881 } }
I20260923 17:28:22.236624 28900 raft_consensus.cc:729] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 97d1965102a7423f87a0de343bdbd507 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97d1965102a7423f87a0de343bdbd507, State: Initialized, Role: FOLLOWER
I20260923 17:28:22.237128 28900 consensus_queue.cc:260] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 97d1965102a7423f87a0de343bdbd507 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d1965102a7423f87a0de343bdbd507" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42517 } } peers { permanent_uuid: "76d05fb754494f3582f4d96f1d3dd5f0" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34709 } } peers { permanent_uuid: "cd15e54eada447f5b9e949c46b37a1b0" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45881 } }
I20260923 17:28:22.238169 28900 ts_tablet_manager.cc:1428] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 97d1965102a7423f87a0de343bdbd507: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260923 17:28:22.238250 29011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:22.237961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.2:56503)
0923 17:28:22.238108 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:22.238173 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260923 17:28:22.238940 29011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:22.238682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.2:40499)
0923 17:28:22.238803 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:22.238859 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260923 17:28:22.355932 29012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:22.356226 29012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:22.356271 29012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:22.360741 29012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:22.360854 29012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:22.360920 29012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.3
I20260923 17:28:22.366384 29012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:41009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.3:34709
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.229.3
--webserver_port=33533
--tserver_master_addrs=127.11.229.62:34709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.3
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:22.367522 29012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:22.368557 29012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:22.378877 29021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:22.378885 29019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:22.378917 29018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:22.379189 29012 server_base.cc:1034] running on GCE node
I20260923 17:28:22.379668 29012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:22.380395 29012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:22.381584 29012 hybrid_clock.cc:648] HybridClock initialized: now 1790184502381557 us; error 40 us; skew 500 ppm
Sep 23 17:28:22 dist-test-slave-ptl9 krb5kdc[27555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184502, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:22.384570 29012 init.cc:377] Logged in from keytab as kudu/127.11.229.3@KRBTEST.COM (short username kudu)
I20260923 17:28:22.385511 29012 webserver.cc:460] Webserver started at http://127.11.229.3:33533/ using document root <none> and password file <none>
I20260923 17:28:22.386057 29012 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:22.386118 29012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:22.390280 29012 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260923 17:28:22.393115 29028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:22.394712 29012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:28:22.394868 29012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/wal
uuid: "76d05fb754494f3582f4d96f1d3dd5f0"
format_stamp: "Formatted at 2026-09-23 17:27:39 on dist-test-slave-ptl9"
server_key: "986939a2dd0d35b6bff2290816021f51"
server_key_iv: "26c775f349b525de743936e442fc1b76"
server_key_version: "encryptionkey@0"
I20260923 17:28:22.395488 29012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:22.420013 29012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:22.425112 29012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:22.425666 29012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:22.426537 29012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:22.427507 29012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:22.427558 29012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:22.427603 29012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:22.427657 29012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:22.438500 29012 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.3:34709
I20260923 17:28:22.438524 29141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.3:34709 every 8 connection(s)
I20260923 17:28:22.439668 29012 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
I20260923 17:28:22.443976 29144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:22.442291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.3:49789)
0923 17:28:22.443081 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:22.443202 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":121,"threads_started":1}
I20260923 17:28:22.444643 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 29012
W20260923 17:28:22.444661 29142 heartbeater.cc:644] Failed to heartbeat to 127.11.229.62:34709 (0 consecutive failures): Network error: Failed to ping master at 127.11.229.62:34709: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
I20260923 17:28:22.445504 29144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:22.445139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.3:41445)
0923 17:28:22.445329 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:22.445410 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260923 17:28:22.446590 29144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:22.446336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.3:33379)
0923 17:28:22.446482 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:22.446530 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260923 17:28:22.446878 29142 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 17:28:23.056602 29146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:23.056066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.1:32815)
0923 17:28:23.056410 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:23.056479 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":89,"threads_started":1}
I20260923 17:28:23.240739 29147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:23.240021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.2:59573)
0923 17:28:23.240507 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:23.240598 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":142,"threads_started":1}
I20260923 17:28:23.419302 29148 raft_consensus.cc:482] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:28:23.419585 29148 raft_consensus.cc:504] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d1965102a7423f87a0de343bdbd507" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42517 } } peers { permanent_uuid: "76d05fb754494f3582f4d96f1d3dd5f0" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34709 } } peers { permanent_uuid: "cd15e54eada447f5b9e949c46b37a1b0" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45881 } }
I20260923 17:28:23.420928 29148 leader_election.cc:288] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97d1965102a7423f87a0de343bdbd507 (127.11.229.2:42517), 76d05fb754494f3582f4d96f1d3dd5f0 (127.11.229.3:34709)
Sep 23 17:28:23 dist-test-slave-ptl9 krb5kdc[27555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184501, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for kudu/127.11.229.3@KRBTEST.COM
Sep 23 17:28:23 dist-test-slave-ptl9 krb5kdc[27555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184501, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for kudu/127.11.229.2@KRBTEST.COM
I20260923 17:28:23.446420 29151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:23.421811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:50259 (local address 127.11.229.3:34709)
0923 17:28:23.422234 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:23.422245 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:23.422284 (+ 39us) server_negotiation.cc:407] Connection header received
0923 17:28:23.423164 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:23.423172 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:23.423595 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:23.423899 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:23.424756 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:23.425931 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:23.427495 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:23.427965 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:23.428134 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:23.437216 (+ 9082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:23.437222 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:23.437238 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:23.437322 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:23.442743 (+ 5421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:23.443628 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:23.443636 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:23.443642 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:23.443861 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:23.444403 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:23.444406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:23.444407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:23.445441 (+ 1034us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:23.445589 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:23.445932 (+ 343us) server_negotiation.cc:299] Negotiation successful
0923 17:28:23.446204 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":89,"threads_started":1}
I20260923 17:28:23.446415 29150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:23.421636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:34709 (local address 127.11.229.1:50259)
0923 17:28:23.421970 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:23.421984 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:23.422087 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:23.424000 (+ 1913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:23.424004 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:23.424263 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:23.424504 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:23.424515 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:23.426145 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:23.426149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:23.427312 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:23.427324 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:23.428106 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:23.428110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:23.428340 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:23.434735 (+ 6395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:23.434757 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:23.436869 (+ 2112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:23.443157 (+ 6288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:23.443160 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:23.443162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:23.443476 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:23.444003 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:23.444007 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:23.444009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:23.444253 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:23.445557 (+ 1304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:23.445560 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:23.445665 (+ 105us) client_negotiation.cc:769] Sending connection context
0923 17:28:23.445801 (+ 136us) client_negotiation.cc:240] Negotiation successful
0923 17:28:23.446201 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":74,"spinlock_wait_cycles":12800,"thread_start_us":64,"threads_started":1}
I20260923 17:28:23.446450 29146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:23.421353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:42517 (local address 127.11.229.1:38261)
0923 17:28:23.421688 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:23.421707 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:23.421827 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:23.423314 (+ 1487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:23.423321 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:23.423858 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:23.424462 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:23.424483 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:23.426595 (+ 2112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:23.426598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:23.427315 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:23.427324 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:23.428138 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:23.428141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:23.428338 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:23.434654 (+ 6316us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:23.434703 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:23.436949 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:23.442908 (+ 5959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:23.442920 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:23.442940 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:23.443453 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:23.444003 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:23.444007 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:23.444009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:23.444253 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:23.445346 (+ 1093us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:23.445352 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:23.445584 (+ 232us) client_negotiation.cc:769] Sending connection context
0923 17:28:23.445794 (+ 210us) client_negotiation.cc:240] Negotiation successful
0923 17:28:23.446294 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":248,"spinlock_wait_cycles":36992}
I20260923 17:28:23.446470 29149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:23.421459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:38261 (local address 127.11.229.2:42517)
0923 17:28:23.421948 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:23.421967 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:23.422021 (+ 54us) server_negotiation.cc:407] Connection header received
0923 17:28:23.422362 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:23.422369 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:23.422894 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:23.423187 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:23.424736 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:23.426422 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:23.427451 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:23.427976 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:23.428164 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:23.437212 (+ 9048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:23.437220 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:23.437238 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:23.437347 (+ 109us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:23.442702 (+ 5355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:23.443628 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:23.443636 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:23.443642 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:23.443851 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:23.444395 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:23.444399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:23.444401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:23.445054 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:23.445216 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:23.445920 (+ 704us) server_negotiation.cc:299] Negotiation successful
0923 17:28:23.446238 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":96,"threads_started":1}
I20260923 17:28:23.447916 29152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:23.447449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.3:51669)
0923 17:28:23.447744 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:23.447800 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":81,"threads_started":1}
I20260923 17:28:23.449434 28963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad77abf6b7c4bebbb7978f0dbdf240a" candidate_uuid: "cd15e54eada447f5b9e949c46b37a1b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "97d1965102a7423f87a0de343bdbd507" is_pre_election: true
W20260923 17:28:23.449700 28759 leader_election.cc:341] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 76d05fb754494f3582f4d96f1d3dd5f0 (127.11.229.3:34709): Not found: Tablet not found: 5ad77abf6b7c4bebbb7978f0dbdf240a
I20260923 17:28:23.449782 28963 raft_consensus.cc:2454] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 97d1965102a7423f87a0de343bdbd507 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd15e54eada447f5b9e949c46b37a1b0 in term 1.
I20260923 17:28:23.450632 28759 leader_election.cc:302] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 97d1965102a7423f87a0de343bdbd507, cd15e54eada447f5b9e949c46b37a1b0; no voters: 76d05fb754494f3582f4d96f1d3dd5f0
I20260923 17:28:23.450999 29148 raft_consensus.cc:2781] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260923 17:28:23.451077 29148 raft_consensus.cc:482] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:28:23.451148 29148 raft_consensus.cc:3037] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0 [term 1 FOLLOWER]: Advancing to term 2
I20260923 17:28:23.453620 29148 raft_consensus.cc:504] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d1965102a7423f87a0de343bdbd507" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42517 } } peers { permanent_uuid: "76d05fb754494f3582f4d96f1d3dd5f0" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34709 } } peers { permanent_uuid: "cd15e54eada447f5b9e949c46b37a1b0" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45881 } }
I20260923 17:28:23.454074 29148 leader_election.cc:288] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0 [CANDIDATE]: Term 2 election: Requested vote from peers 97d1965102a7423f87a0de343bdbd507 (127.11.229.2:42517), 76d05fb754494f3582f4d96f1d3dd5f0 (127.11.229.3:34709)
I20260923 17:28:23.454810 28963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad77abf6b7c4bebbb7978f0dbdf240a" candidate_uuid: "cd15e54eada447f5b9e949c46b37a1b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "97d1965102a7423f87a0de343bdbd507"
I20260923 17:28:23.455015 28963 raft_consensus.cc:3037] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 97d1965102a7423f87a0de343bdbd507 [term 1 FOLLOWER]: Advancing to term 2
W20260923 17:28:23.455287 28759 leader_election.cc:341] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 76d05fb754494f3582f4d96f1d3dd5f0 (127.11.229.3:34709): Not found: Tablet not found: 5ad77abf6b7c4bebbb7978f0dbdf240a
I20260923 17:28:23.457764 28963 raft_consensus.cc:2454] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 97d1965102a7423f87a0de343bdbd507 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd15e54eada447f5b9e949c46b37a1b0 in term 2.
I20260923 17:28:23.458185 28759 leader_election.cc:302] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 97d1965102a7423f87a0de343bdbd507, cd15e54eada447f5b9e949c46b37a1b0; no voters: 76d05fb754494f3582f4d96f1d3dd5f0
I20260923 17:28:23.458451 29148 raft_consensus.cc:2781] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 17:28:23.458705 29148 raft_consensus.cc:686] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0 [term 2 LEADER]: Becoming Leader. State: Replica: cd15e54eada447f5b9e949c46b37a1b0, State: Running, Role: LEADER
I20260923 17:28:23.459098 29148 consensus_queue.cc:237] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d1965102a7423f87a0de343bdbd507" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42517 } } peers { permanent_uuid: "76d05fb754494f3582f4d96f1d3dd5f0" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34709 } } peers { permanent_uuid: "cd15e54eada447f5b9e949c46b37a1b0" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45881 } }
I20260923 17:28:23.459453 29146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:23.459192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.1:58573)
0923 17:28:23.459329 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:23.459387 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260923 17:28:23.879092 28759 consensus_peers.cc:487] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0 -> Peer 76d05fb754494f3582f4d96f1d3dd5f0 (127.11.229.3:34709): Couldn't send request to peer 76d05fb754494f3582f4d96f1d3dd5f0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ad77abf6b7c4bebbb7978f0dbdf240a. This is attempt 1: this message will repeat every 5th retry.
I20260923 17:28:23.990059 28963 raft_consensus.cc:1261] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 97d1965102a7423f87a0de343bdbd507 [term 2 FOLLOWER]: Refusing update from remote peer cd15e54eada447f5b9e949c46b37a1b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260923 17:28:23.991366 29148 consensus_queue.cc:1035] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97d1965102a7423f87a0de343bdbd507" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260923 17:28:23.994354 29156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:23.993850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.2:46841)
0923 17:28:23.994141 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:23.994212 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":66,"threads_started":1}
I20260923 17:28:23.998862 29156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:23.998584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.2:50215)
0923 17:28:23.998723 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:23.998779 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260923 17:28:23.998899 29161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:23.998342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.1:55021)
0923 17:28:23.998717 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:23.998792 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":92,"threads_started":1}
I20260923 17:28:24.323907 29162 ts_tablet_manager.cc:927] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0: Initiating tablet copy from peer cd15e54eada447f5b9e949c46b37a1b0 (127.11.229.1:45881)
I20260923 17:28:24.324743 29162 tablet_copy_client.cc:316] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0: tablet copy: Beginning tablet copy session from remote peer at address 127.11.229.1:45881
Sep 23 17:28:24 dist-test-slave-ptl9 krb5kdc[27555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184502, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for kudu/127.11.229.1@KRBTEST.COM
I20260923 17:28:24.338059 29164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:24.325548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:45881 (local address 127.11.229.3:50403)
0923 17:28:24.325933 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:24.325957 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:24.326134 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:24.326787 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:24.326796 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:24.327208 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:24.327373 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:24.327381 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:24.328476 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:24.328479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:24.329189 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:24.329200 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:24.330011 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:24.330016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:24.330190 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:24.331087 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:24.331113 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:24.332837 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:24.335541 (+ 2704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:24.335555 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:24.335567 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:24.335917 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:24.336307 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:24.336311 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:24.336313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:24.336430 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:24.337168 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:24.337176 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:24.337407 (+ 231us) client_negotiation.cc:769] Sending connection context
0923 17:28:24.337578 (+ 171us) client_negotiation.cc:240] Negotiation successful
0923 17:28:24.337828 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":120,"threads_started":1}
I20260923 17:28:24.338052 29165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:24.325732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:50403 (local address 127.11.229.1:45881)
0923 17:28:24.326274 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:24.326293 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:24.326320 (+ 27us) server_negotiation.cc:407] Connection header received
0923 17:28:24.326380 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:24.326385 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:24.326582 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:24.326714 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:24.327534 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:24.328332 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:24.329392 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:24.329793 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:24.329897 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:24.333238 (+ 3341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:24.333282 (+ 44us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:24.333299 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:24.333347 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:24.335363 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:24.336096 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:24.336102 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:24.336103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:24.336198 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:24.336537 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:24.336542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:24.336544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:24.337021 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:24.337199 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:24.337677 (+ 478us) server_negotiation.cc:299] Negotiation successful
0923 17:28:24.337856 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":134,"threads_started":1}
I20260923 17:28:24.339362 28847 tablet_copy_service.cc:140] P cd15e54eada447f5b9e949c46b37a1b0: Received BeginTabletCopySession request for tablet 5ad77abf6b7c4bebbb7978f0dbdf240a from peer 76d05fb754494f3582f4d96f1d3dd5f0 ({username='kudu', principal='kudu/127.11.229.3@KRBTEST.COM'} at 127.11.229.3:50403)
I20260923 17:28:24.339540 28847 tablet_copy_service.cc:161] P cd15e54eada447f5b9e949c46b37a1b0: Beginning new tablet copy session on tablet 5ad77abf6b7c4bebbb7978f0dbdf240a from peer 76d05fb754494f3582f4d96f1d3dd5f0 at {username='kudu', principal='kudu/127.11.229.3@KRBTEST.COM'} at 127.11.229.3:50403: session id = 76d05fb754494f3582f4d96f1d3dd5f0-5ad77abf6b7c4bebbb7978f0dbdf240a
I20260923 17:28:24.340794 28847 tablet_copy_source_session.cc:205] T 5ad77abf6b7c4bebbb7978f0dbdf240a P cd15e54eada447f5b9e949c46b37a1b0: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 17:28:24.345589 29162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ad77abf6b7c4bebbb7978f0dbdf240a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:24.353248 29162 tablet_copy_client.cc:791] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0: tablet copy: Starting download of 0 data blocks...
I20260923 17:28:24.353590 29162 tablet_copy_client.cc:655] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0: tablet copy: Starting download of 1 WAL segments...
I20260923 17:28:24.356016 29162 tablet_copy_client.cc:531] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 17:28:24.358604 29162 tablet_bootstrap.cc:492] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0: Bootstrap starting.
I20260923 17:28:24.365685 29162 log.cc:824] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:24.374254 29162 tablet_bootstrap.cc:492] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:28:24.374671 29162 tablet_bootstrap.cc:492] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0: Bootstrap complete.
I20260923 17:28:24.375083 29162 ts_tablet_manager.cc:1397] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0: Time spent bootstrapping tablet: real 0.017s user 0.016s sys 0.000s
I20260923 17:28:24.377838 29162 raft_consensus.cc:348] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d1965102a7423f87a0de343bdbd507" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42517 } } peers { permanent_uuid: "76d05fb754494f3582f4d96f1d3dd5f0" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34709 } } peers { permanent_uuid: "cd15e54eada447f5b9e949c46b37a1b0" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45881 } }
I20260923 17:28:24.378191 29162 raft_consensus.cc:729] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d05fb754494f3582f4d96f1d3dd5f0, State: Initialized, Role: FOLLOWER
I20260923 17:28:24.378737 29162 consensus_queue.cc:260] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d1965102a7423f87a0de343bdbd507" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 42517 } } peers { permanent_uuid: "76d05fb754494f3582f4d96f1d3dd5f0" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34709 } } peers { permanent_uuid: "cd15e54eada447f5b9e949c46b37a1b0" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45881 } }
I20260923 17:28:24.379998 29162 ts_tablet_manager.cc:1428] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260923 17:28:24.380386 29164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:24.380032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.3:60155)
0923 17:28:24.380242 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:24.380305 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260923 17:28:24.380861 28847 tablet_copy_service.cc:342] P cd15e54eada447f5b9e949c46b37a1b0: Request end of tablet copy session 76d05fb754494f3582f4d96f1d3dd5f0-5ad77abf6b7c4bebbb7978f0dbdf240a received from {username='kudu', principal='kudu/127.11.229.3@KRBTEST.COM'} at 127.11.229.3:50403
I20260923 17:28:24.381006 28847 tablet_copy_service.cc:433] P cd15e54eada447f5b9e949c46b37a1b0: ending tablet copy session 76d05fb754494f3582f4d96f1d3dd5f0-5ad77abf6b7c4bebbb7978f0dbdf240a on tablet 5ad77abf6b7c4bebbb7978f0dbdf240a with peer 76d05fb754494f3582f4d96f1d3dd5f0
I20260923 17:28:24.381165 29164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:24.380896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.3:59267)
0923 17:28:24.381031 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:24.381088 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260923 17:28:24.893476 29169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:24.893010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.1:55681)
0923 17:28:24.893247 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:24.893374 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":80,"threads_started":1}
I20260923 17:28:24.893445 29096 raft_consensus.cc:1203] T 5ad77abf6b7c4bebbb7978f0dbdf240a P 76d05fb754494f3582f4d96f1d3dd5f0 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260923 17:28:24.895248 29171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:24.894843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.3:33771)
0923 17:28:24.895076 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:24.895140 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":68,"threads_started":1}
I20260923 17:28:25.000221 29172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:24.999639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.2:52269)
0923 17:28:25.000049 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:25.000117 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":109,"threads_started":1}
I20260923 17:28:25.895153 29173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:25.894533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.1:48805)
0923 17:28:25.894955 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:25.895044 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":108,"threads_started":1}
I20260923 17:28:25.896592 29174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:25.896130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.3:44857)
0923 17:28:25.896439 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:25.896500 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":105,"threads_started":1}
I20260923 17:28:26.001971 29175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:26.001377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.2:44131)
0923 17:28:26.001778 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:26.001853 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":98,"threads_started":1}
I20260923 17:28:26.896864 29176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:26.896343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.1:45155)
0923 17:28:26.896701 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:26.896757 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":146,"threads_started":1}
I20260923 17:28:26.897861 29177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:26.897447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.3:53489)
0923 17:28:26.897719 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:26.897773 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":66,"threads_started":1}
I20260923 17:28:27.003615 29178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:27.003079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.2:38759)
0923 17:28:27.003436 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:27.003508 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":100,"threads_started":1}
I20260923 17:28:27.898520 29179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:27.898051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.1:46053)
0923 17:28:27.898353 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:27.898418 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":78,"threads_started":1}
I20260923 17:28:27.899382 29179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:27.899108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.1:41553)
0923 17:28:27.899252 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:27.899306 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260923 17:28:27.899487 29180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:27.898845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.3:50509)
0923 17:28:27.899301 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:27.899383 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":103,"threads_started":1}
I20260923 17:28:28.005441 29181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:28.004859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.2:50221)
0923 17:28:28.005185 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:28.005256 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":106,"threads_started":1}
I20260923 17:28:28.900870 29182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:28.900332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.1:53677)
0923 17:28:28.900677 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:28.900756 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":89,"threads_started":1}
I20260923 17:28:28.901023 29183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:28.900405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.3:58189)
0923 17:28:28.900798 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:28.900856 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":103,"threads_started":1}
I20260923 17:28:29.007503 29184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:29.006788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.2:45753)
0923 17:28:29.007203 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:29.007302 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":96,"threads_started":1}
I20260923 17:28:29.902343 29186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:29.901854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.3:34149)
0923 17:28:29.902122 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:29.902216 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":76,"threads_started":1}
I20260923 17:28:29.902343 29185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:29.901743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.1:36415)
0923 17:28:29.902151 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:29.902214 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":95,"threads_started":1}
I20260923 17:28:30.009711 29187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:30.008987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.2:40095)
0923 17:28:30.009429 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:30.009511 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":123,"threads_started":1}
I20260923 17:28:30.904270 29188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:30.903613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.3:58491)
0923 17:28:30.904074 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:30.904151 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":113,"threads_started":1}
I20260923 17:28:30.904285 29189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:30.903613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.1:34929)
0923 17:28:30.904041 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:30.904142 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":112,"threads_started":1}
I20260923 17:28:31.012004 29190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:31.011317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.2:44257)
0923 17:28:31.011775 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:31.011874 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":121,"threads_started":1}
I20260923 17:28:31.906105 29191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:31.905368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.3:40073)
0923 17:28:31.905849 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:31.905958 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":116,"threads_started":1}
I20260923 17:28:31.906155 29192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:31.905541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.1:55793)
0923 17:28:31.905877 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:31.905960 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":78,"threads_started":1}
I20260923 17:28:32.014112 29193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:32.013382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.2:58563)
0923 17:28:32.013893 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:32.013993 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":164,"threads_started":1}
I20260923 17:28:32.907590 29194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:32.907099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.3:56475)
0923 17:28:32.907428 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:32.907490 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":97,"threads_started":1}
I20260923 17:28:32.907635 29195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:32.907228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.1:43701)
0923 17:28:32.907482 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:32.907536 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":76,"threads_started":1}
I20260923 17:28:33.015612 29196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:33.015140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.2:35757)
0923 17:28:33.015439 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:33.015514 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":92,"threads_started":1}
I20260923 17:28:33.908949 29199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:33.908448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.3:41513)
0923 17:28:33.908804 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:33.908861 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":123,"threads_started":1}
I20260923 17:28:33.908943 29198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:33.908446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.1:54853)
0923 17:28:33.908765 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:33.908842 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":91,"threads_started":1}
I20260923 17:28:34.016830 29200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:34.016428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.2:34089)
0923 17:28:34.016670 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:34.016727 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":68,"threads_started":1}
I20260923 17:28:34.910509 29202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:34.910062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.3:41511)
0923 17:28:34.910334 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:34.910391 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":72,"threads_started":1}
I20260923 17:28:34.910509 29201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:34.909957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.1:35357)
0923 17:28:34.910323 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:34.910391 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":97,"threads_started":1}
I20260923 17:28:35.018174 29203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:35.017719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.2:47347)
0923 17:28:35.018002 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:35.018076 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":84,"threads_started":1}
I20260923 17:28:35.911765 29204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:35.911311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.1:51505)
0923 17:28:35.911614 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:35.911667 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":99,"threads_started":1}
I20260923 17:28:35.911859 29205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:35.911360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.3:40765)
0923 17:28:35.911674 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:35.911729 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":96,"threads_started":1}
I20260923 17:28:36.019433 29206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:36.019000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.2:59653)
0923 17:28:36.019268 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:36.019332 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":89,"threads_started":1}
I20260923 17:28:36.913136 29208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:36.912679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.1:34337)
0923 17:28:36.912977 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:36.913039 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":91,"threads_started":1}
I20260923 17:28:36.913172 29207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:36.912679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.3:38603)
0923 17:28:36.913038 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:36.913089 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":97,"threads_started":1}
I20260923 17:28:37.020768 29209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:37.020214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34709 (local address 127.11.229.2:47353)
0923 17:28:37.020509 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:37.020617 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.62:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":78,"threads_started":1}
I20260923 17:28:37.448719 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 28702
I20260923 17:28:37.455567 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 28876
I20260923 17:28:37.461594 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 29012
2026-09-23T17:28:37Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59824 ms)
[----------] 30 tests from SecurityITest (245857 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:28:37 dist-test-slave-ptl9 krb5kdc[29212](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:28:37 dist-test-slave-ptl9 krb5kdc[29212](info): set up 2 sockets
Sep 23 17:28:37 dist-test-slave-ptl9 krb5kdc[29212](info): commencing operation
krb5kdc: starting...
W20260923 17:28:38.533706 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:28:38 dist-test-slave-ptl9 krb5kdc[29212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:28:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:28:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.229.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
I20260923 17:28:38.670240 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:40791
--webserver_interface=127.11.229.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:41607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:40791
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:38.790488 29225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:38.790848 29225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:38.790928 29225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:38.795513 29225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:28:38.795670 29225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:38.795713 29225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:38.795753 29225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:28:38.795790 29225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:28:38.802002 29225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:41607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:40791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:40791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:38.803474 29225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:38.804687 29225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 17:28:38.815325 29225 server_base.cc:1034] running on GCE node
W20260923 17:28:38.815272 29233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:38.815272 29230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:38.815474 29231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:38.816248 29225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:38.817647 29225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:38.819254 29225 hybrid_clock.cc:648] HybridClock initialized: now 1790184518819230 us; error 38 us; skew 500 ppm
Sep 23 17:28:38 dist-test-slave-ptl9 krb5kdc[29212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184518, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:38.822872 29225 init.cc:377] Logged in from keytab as kudu/127.11.229.62@KRBTEST.COM (short username kudu)
I20260923 17:28:38.823962 29225 webserver.cc:460] Webserver started at http://127.11.229.62:37911/ using document root <none> and password file <none>
I20260923 17:28:38.824586 29225 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:38.824679 29225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:38.824932 29225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:38.827317 29225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "21490c5f8c4f4c5fadb278de71a851d5"
format_stamp: "Formatted at 2026-09-23 17:28:38 on dist-test-slave-ptl9"
server_key: "c8e4f912c832f807aead06f162b7b257"
server_key_iv: "8adb7509951542bdc2957d6f4be2a5ec"
server_key_version: "encryptionkey@0"
I20260923 17:28:38.827931 29225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "21490c5f8c4f4c5fadb278de71a851d5"
format_stamp: "Formatted at 2026-09-23 17:28:38 on dist-test-slave-ptl9"
server_key: "c8e4f912c832f807aead06f162b7b257"
server_key_iv: "8adb7509951542bdc2957d6f4be2a5ec"
server_key_version: "encryptionkey@0"
I20260923 17:28:38.832258 29225 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 17:28:38.835301 29240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:38.837399 29225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:28:38.837658 29225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "21490c5f8c4f4c5fadb278de71a851d5"
format_stamp: "Formatted at 2026-09-23 17:28:38 on dist-test-slave-ptl9"
server_key: "c8e4f912c832f807aead06f162b7b257"
server_key_iv: "8adb7509951542bdc2957d6f4be2a5ec"
server_key_version: "encryptionkey@0"
I20260923 17:28:38.837859 29225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:38.855242 29225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:38.859030 29225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:38.859283 29225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:38.868150 29225 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:40791
I20260923 17:28:38.868181 29292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:40791 every 8 connection(s)
I20260923 17:28:38.869199 29225 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:28:38.872278 29293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:38.878392 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 29225
I20260923 17:28:38.878548 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:28:38.878868 12180 external_mini_cluster.cc:1428] Setting key e2ced338e218d22d84872cdb489d987d
I20260923 17:28:38.878867 29293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21490c5f8c4f4c5fadb278de71a851d5: Bootstrap starting.
I20260923 17:28:38.882001 29293 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21490c5f8c4f4c5fadb278de71a851d5: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:38.882799 29293 log.cc:824] T 00000000000000000000000000000000 P 21490c5f8c4f4c5fadb278de71a851d5: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:38.885315 29293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21490c5f8c4f4c5fadb278de71a851d5: No bootstrap required, opened a new log
Sep 23 17:28:38 dist-test-slave-ptl9 krb5kdc[29212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:28:38.890015 29293 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21490c5f8c4f4c5fadb278de71a851d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21490c5f8c4f4c5fadb278de71a851d5" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 40791 } }
I20260923 17:28:38.890359 29293 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21490c5f8c4f4c5fadb278de71a851d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:38.890446 29293 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21490c5f8c4f4c5fadb278de71a851d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21490c5f8c4f4c5fadb278de71a851d5, State: Initialized, Role: FOLLOWER
I20260923 17:28:38.891008 29293 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21490c5f8c4f4c5fadb278de71a851d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21490c5f8c4f4c5fadb278de71a851d5" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 40791 } }
I20260923 17:28:38.891144 29293 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21490c5f8c4f4c5fadb278de71a851d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:28:38.891237 29293 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21490c5f8c4f4c5fadb278de71a851d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:28:38.891381 29293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21490c5f8c4f4c5fadb278de71a851d5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:38.892796 29293 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21490c5f8c4f4c5fadb278de71a851d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21490c5f8c4f4c5fadb278de71a851d5" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 40791 } }
I20260923 17:28:38.893203 29293 leader_election.cc:302] T 00000000000000000000000000000000 P 21490c5f8c4f4c5fadb278de71a851d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21490c5f8c4f4c5fadb278de71a851d5; no voters:
I20260923 17:28:38.893600 29293 leader_election.cc:288] T 00000000000000000000000000000000 P 21490c5f8c4f4c5fadb278de71a851d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:28:38.893718 29298 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21490c5f8c4f4c5fadb278de71a851d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:38.893971 29298 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21490c5f8c4f4c5fadb278de71a851d5 [term 1 LEADER]: Becoming Leader. State: Replica: 21490c5f8c4f4c5fadb278de71a851d5, State: Running, Role: LEADER
I20260923 17:28:38.894407 29298 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21490c5f8c4f4c5fadb278de71a851d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21490c5f8c4f4c5fadb278de71a851d5" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 40791 } }
I20260923 17:28:38.895112 29293 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21490c5f8c4f4c5fadb278de71a851d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:28:38.896082 29298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21490c5f8c4f4c5fadb278de71a851d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21490c5f8c4f4c5fadb278de71a851d5. Latest consensus state: current_term: 1 leader_uuid: "21490c5f8c4f4c5fadb278de71a851d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21490c5f8c4f4c5fadb278de71a851d5" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 40791 } } }
I20260923 17:28:38.896287 29298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21490c5f8c4f4c5fadb278de71a851d5 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:38.896525 29299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21490c5f8c4f4c5fadb278de71a851d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21490c5f8c4f4c5fadb278de71a851d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21490c5f8c4f4c5fadb278de71a851d5" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 40791 } } }
I20260923 17:28:38.896589 29296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:38.880469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55192 (local address 127.11.229.62:40791)
0923 17:28:38.881002 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:38.881018 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:38.881065 (+ 47us) server_negotiation.cc:407] Connection header received
0923 17:28:38.881839 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:38.881859 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:38.882267 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:38.882644 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:38.883531 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:38.885032 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:38.886094 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:38.886794 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:38.886977 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:38.889897 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:38.889904 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:38.889923 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:38.889958 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:38.892080 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:38.892666 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:38.892672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:38.892682 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:38.892814 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:38.893169 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:38.893175 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:38.893178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:38.893849 (+ 671us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:38.894721 (+ 872us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:38.895126 (+ 405us) server_negotiation.cc:299] Negotiation successful
0923 17:28:38.895500 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":149,"threads_started":1}
I20260923 17:28:38.896627 29299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21490c5f8c4f4c5fadb278de71a851d5 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:38.896739 29307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:28:38.900177 29307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:28:38.907096 29307 catalog_manager.cc:1347] Generated new cluster ID: 7dd244b6eb394798bef4b6192717f9ed
I20260923 17:28:38.907229 29307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:28:38.929085 29307 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:28:38.930212 29307 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:28:38.937090 29307 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21490c5f8c4f4c5fadb278de71a851d5: Generated new TSK 0
I20260923 17:28:38.938046 29307 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.229.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab.
Entry for principal kudu/127.11.229.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab.
I20260923 17:28:38.982020 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.1:0
--local_ip_for_outbound_sockets=127.11.229.1
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:40791
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:41607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:39.100581 29319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:39.100896 29319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:39.100986 29319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:39.105163 29319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:39.105258 29319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:39.105383 29319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.1
I20260923 17:28:39.110939 29319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:41607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:40791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.1
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:39.112062 29319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:39.113132 29319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:39.123775 29324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:39.123790 29325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:39.123767 29327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:39.124135 29319 server_base.cc:1034] running on GCE node
I20260923 17:28:39.124581 29319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:39.125358 29319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:39.126547 29319 hybrid_clock.cc:648] HybridClock initialized: now 1790184519126509 us; error 51 us; skew 500 ppm
Sep 23 17:28:39 dist-test-slave-ptl9 krb5kdc[29212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184519, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:39.129528 29319 init.cc:377] Logged in from keytab as kudu/127.11.229.1@KRBTEST.COM (short username kudu)
I20260923 17:28:39.130477 29319 webserver.cc:460] Webserver started at http://127.11.229.1:39633/ using document root <none> and password file <none>
I20260923 17:28:39.131083 29319 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:39.131174 29319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:39.131417 29319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:39.133634 29319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-0/data/instance:
uuid: "9768fd85be8e4efbb467ec3635422757"
format_stamp: "Formatted at 2026-09-23 17:28:39 on dist-test-slave-ptl9"
server_key: "f055151c67f1bb56045535551c21fe9e"
server_key_iv: "1fd6d68a03cb07751711c611f4d993f9"
server_key_version: "encryptionkey@0"
I20260923 17:28:39.134230 29319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance:
uuid: "9768fd85be8e4efbb467ec3635422757"
format_stamp: "Formatted at 2026-09-23 17:28:39 on dist-test-slave-ptl9"
server_key: "f055151c67f1bb56045535551c21fe9e"
server_key_iv: "1fd6d68a03cb07751711c611f4d993f9"
server_key_version: "encryptionkey@0"
I20260923 17:28:39.138108 29319 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:28:39.140723 29334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:39.142100 29319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:28:39.142242 29319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-0/wal
uuid: "9768fd85be8e4efbb467ec3635422757"
format_stamp: "Formatted at 2026-09-23 17:28:39 on dist-test-slave-ptl9"
server_key: "f055151c67f1bb56045535551c21fe9e"
server_key_iv: "1fd6d68a03cb07751711c611f4d993f9"
server_key_version: "encryptionkey@0"
I20260923 17:28:39.142347 29319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:39.156945 29319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:39.160205 29319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:39.160418 29319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:39.161026 29319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:39.161934 29319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:39.161983 29319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:39.162024 29319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:39.162077 29319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:39.171191 29319 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.1:38321
I20260923 17:28:39.171221 29447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.1:38321 every 8 connection(s)
I20260923 17:28:39.172132 29319 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
I20260923 17:28:39.179615 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 29319
I20260923 17:28:39.179777 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance
I20260923 17:28:39.180087 12180 external_mini_cluster.cc:1428] Setting key da7f3f364ddb917c2e7f1f7f360bd4b4
Sep 23 17:28:39 dist-test-slave-ptl9 krb5kdc[29212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184519, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:28:39.187237 29296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:39.174059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:54287 (local address 127.11.229.62:40791)
0923 17:28:39.174201 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:39.174205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:39.175279 (+ 1074us) server_negotiation.cc:407] Connection header received
0923 17:28:39.176195 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:39.176199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:39.176259 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:39.176343 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:39.177619 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:39.178178 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:39.179363 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:39.179600 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:39.179717 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:39.182727 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:39.182731 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:39.182733 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:39.182757 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:39.184574 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:39.185164 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:39.185171 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:39.185176 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:39.185296 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:39.185647 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:39.185653 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:39.185658 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:39.185939 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:39.186048 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:39.186813 (+ 765us) server_negotiation.cc:299] Negotiation successful
0923 17:28:39.187001 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 17:28:39.187992 29450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:39.174398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:40791 (local address 127.11.229.1:54287)
0923 17:28:39.175113 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:39.175157 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:39.175986 (+ 829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:39.176517 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:39.176524 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:39.176892 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:39.177466 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:39.177483 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:39.178321 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:39.178325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:39.179210 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:39.179229 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:39.179769 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:39.179773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:39.179988 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:39.180717 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:39.180743 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:39.182545 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:39.184707 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:39.184713 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:39.184757 (+ 44us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:39.185075 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:39.185396 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:39.185398 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:39.185399 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:39.185552 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:39.186093 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:39.186098 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:39.186548 (+ 450us) client_negotiation.cc:769] Sending connection context
0923 17:28:39.186726 (+ 178us) client_negotiation.cc:240] Negotiation successful
0923 17:28:39.187018 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":121,"threads_started":1}
I20260923 17:28:39.189432 29448 heartbeater.cc:347] Connected to a master server at 127.11.229.62:40791
I20260923 17:28:39.189710 29448 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:39.190260 29448 heartbeater.cc:511] Master 127.11.229.62:40791 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.229.2@KRBTEST.COM; defaulting to no policy
I20260923 17:28:39.191978 29257 ts_manager.cc:194] Registered new tserver with Master: 9768fd85be8e4efbb467ec3635422757 (127.11.229.1:38321)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.2@KRBTEST.COM" created.
I20260923 17:28:39.193238 29257 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.1@KRBTEST.COM'} at 127.11.229.1:54287
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab.
Entry for principal kudu/127.11.229.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab.
I20260923 17:28:39.208781 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.2:0
--local_ip_for_outbound_sockets=127.11.229.2
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:40791
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:41607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:39.330358 29453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:39.330677 29453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:39.330725 29453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:39.335610 29453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:39.335690 29453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:39.335793 29453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.2
I20260923 17:28:39.341624 29453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:41607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:40791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.2
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:39.342952 29453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:39.344242 29453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:39.354763 29459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:39.354784 29458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:39.354769 29461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:39.354928 29453 server_base.cc:1034] running on GCE node
I20260923 17:28:39.355623 29453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:39.356325 29453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:39.357491 29453 hybrid_clock.cc:648] HybridClock initialized: now 1790184519357476 us; error 28 us; skew 500 ppm
Sep 23 17:28:39 dist-test-slave-ptl9 krb5kdc[29212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184519, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:39.360260 29453 init.cc:377] Logged in from keytab as kudu/127.11.229.2@KRBTEST.COM (short username kudu)
I20260923 17:28:39.361240 29453 webserver.cc:460] Webserver started at http://127.11.229.2:37097/ using document root <none> and password file <none>
I20260923 17:28:39.362017 29453 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:39.362110 29453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:39.362349 29453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:39.364523 29453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-1/data/instance:
uuid: "5930b16495e748388349463fe3b22f80"
format_stamp: "Formatted at 2026-09-23 17:28:39 on dist-test-slave-ptl9"
server_key: "9155e6fd0abb1ade5c77e1bb7afe3584"
server_key_iv: "c9e55e990bfe9fcdea258b841eafc4c4"
server_key_version: "encryptionkey@0"
I20260923 17:28:39.365126 29453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance:
uuid: "5930b16495e748388349463fe3b22f80"
format_stamp: "Formatted at 2026-09-23 17:28:39 on dist-test-slave-ptl9"
server_key: "9155e6fd0abb1ade5c77e1bb7afe3584"
server_key_iv: "c9e55e990bfe9fcdea258b841eafc4c4"
server_key_version: "encryptionkey@0"
I20260923 17:28:39.369061 29453 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260923 17:28:39.371765 29468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:39.373091 29453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:28:39.373296 29453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-1/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-1/wal
uuid: "5930b16495e748388349463fe3b22f80"
format_stamp: "Formatted at 2026-09-23 17:28:39 on dist-test-slave-ptl9"
server_key: "9155e6fd0abb1ade5c77e1bb7afe3584"
server_key_iv: "c9e55e990bfe9fcdea258b841eafc4c4"
server_key_version: "encryptionkey@0"
I20260923 17:28:39.373432 29453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:39.396849 29453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:39.400453 29453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:39.400709 29453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:39.401530 29453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:39.402655 29453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:39.402750 29453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:39.402984 29453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:39.403265 29453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:39.413494 29453 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.2:40991
I20260923 17:28:39.413522 29581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.2:40991 every 8 connection(s)
I20260923 17:28:39.414515 29453 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
I20260923 17:28:39.416453 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 29453
I20260923 17:28:39.416587 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance
I20260923 17:28:39.416921 12180 external_mini_cluster.cc:1428] Setting key bb7fccd7209130f4765dcb9150d41fae
Sep 23 17:28:39 dist-test-slave-ptl9 krb5kdc[29212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184519, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.229.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.3@KRBTEST.COM" created.
I20260923 17:28:39.431183 29296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:39.416489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:36829 (local address 127.11.229.62:40791)
0923 17:28:39.416640 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:39.416646 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:39.417690 (+ 1044us) server_negotiation.cc:407] Connection header received
0923 17:28:39.419091 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:39.419097 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:39.419161 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:39.419256 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:39.420478 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:39.421024 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:39.422427 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:39.422668 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:39.422776 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:39.425887 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:39.425892 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:39.425895 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:39.425922 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:39.428023 (+ 2101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:39.428922 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:39.428926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:39.428928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:39.429172 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:39.429755 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:39.429758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:39.429760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:39.430036 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:39.430172 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:39.430752 (+ 580us) server_negotiation.cc:299] Negotiation successful
0923 17:28:39.430922 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 17:28:39.431910 29584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:39.416946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:40791 (local address 127.11.229.2:36829)
0923 17:28:39.417535 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:39.417592 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:39.418866 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:39.419441 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:39.419448 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:39.419815 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:39.420319 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:39.420333 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:39.421159 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:39.421164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:39.422287 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:39.422299 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:39.422841 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:39.422846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:39.423117 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:39.423813 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:39.423831 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:39.425673 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:39.428199 (+ 2526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:39.428207 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:39.428233 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:39.428673 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:39.429351 (+ 678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:39.429360 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:39.429362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:39.429592 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:39.430175 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:39.430182 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:39.430468 (+ 286us) client_negotiation.cc:769] Sending connection context
0923 17:28:39.430638 (+ 170us) client_negotiation.cc:240] Negotiation successful
0923 17:28:39.430945 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":140,"threads_started":1}
I20260923 17:28:39.433245 29582 heartbeater.cc:347] Connected to a master server at 127.11.229.62:40791
I20260923 17:28:39.433543 29582 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:39.434195 29582 heartbeater.cc:511] Master 127.11.229.62:40791 requested a full tablet report, sending...
I20260923 17:28:39.435513 29257 ts_manager.cc:194] Registered new tserver with Master: 5930b16495e748388349463fe3b22f80 (127.11.229.2:40991)
I20260923 17:28:39.436136 29257 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.2@KRBTEST.COM'} at 127.11.229.2:36829
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab.
Entry for principal kudu/127.11.229.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab.
I20260923 17:28:39.448240 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.3:0
--local_ip_for_outbound_sockets=127.11.229.3
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:40791
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:41607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:39.566421 29587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:39.566730 29587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:39.566804 29587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:39.570927 29587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:39.571012 29587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:39.571105 29587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.3
I20260923 17:28:39.576531 29587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:41607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:40791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.3
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:39.577711 29587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:39.578783 29587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:39.588272 29592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:39.588349 29593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:39.588517 29587 server_base.cc:1034] running on GCE node
W20260923 17:28:39.588371 29595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:39.589360 29587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:39.589994 29587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:39.591162 29587 hybrid_clock.cc:648] HybridClock initialized: now 1790184519591122 us; error 52 us; skew 500 ppm
Sep 23 17:28:39 dist-test-slave-ptl9 krb5kdc[29212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184519, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:39.593982 29587 init.cc:377] Logged in from keytab as kudu/127.11.229.3@KRBTEST.COM (short username kudu)
I20260923 17:28:39.594892 29587 webserver.cc:460] Webserver started at http://127.11.229.3:43941/ using document root <none> and password file <none>
I20260923 17:28:39.595502 29587 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:39.595582 29587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:39.595837 29587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:39.598121 29587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-2/data/instance:
uuid: "127514e6d1c54c60995e114316b8ad1a"
format_stamp: "Formatted at 2026-09-23 17:28:39 on dist-test-slave-ptl9"
server_key: "91446d82052449c07bdff0ba22821b01"
server_key_iv: "46ae53b17a59cefebfc0607e6e687a78"
server_key_version: "encryptionkey@0"
I20260923 17:28:39.598711 29587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance:
uuid: "127514e6d1c54c60995e114316b8ad1a"
format_stamp: "Formatted at 2026-09-23 17:28:39 on dist-test-slave-ptl9"
server_key: "91446d82052449c07bdff0ba22821b01"
server_key_iv: "46ae53b17a59cefebfc0607e6e687a78"
server_key_version: "encryptionkey@0"
I20260923 17:28:39.602579 29587 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:28:39.605072 29602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:39.606371 29587 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:28:39.606467 29587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-2/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-2/wal
uuid: "127514e6d1c54c60995e114316b8ad1a"
format_stamp: "Formatted at 2026-09-23 17:28:39 on dist-test-slave-ptl9"
server_key: "91446d82052449c07bdff0ba22821b01"
server_key_iv: "46ae53b17a59cefebfc0607e6e687a78"
server_key_version: "encryptionkey@0"
I20260923 17:28:39.606554 29587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:39.628626 29587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:39.632066 29587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:39.632234 29587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:39.632809 29587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:39.633728 29587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:39.633774 29587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:39.633816 29587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:39.633831 29587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:39.644707 29587 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.3:33029
I20260923 17:28:39.644740 29715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.3:33029 every 8 connection(s)
I20260923 17:28:39.646332 29587 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
I20260923 17:28:39.656229 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 29587
I20260923 17:28:39.656401 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance
I20260923 17:28:39.656723 12180 external_mini_cluster.cc:1428] Setting key bb6e47a82f0e63ea51f5da9008a8312b
Sep 23 17:28:39 dist-test-slave-ptl9 krb5kdc[29212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184519, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:28:39.662954 29296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:39.649524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:60531 (local address 127.11.229.62:40791)
0923 17:28:39.649735 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:39.649739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:39.650801 (+ 1062us) server_negotiation.cc:407] Connection header received
0923 17:28:39.651696 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:39.651698 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:39.651752 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:39.651867 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:39.653079 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:39.653755 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:39.655311 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:39.655819 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:39.656136 (+ 317us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:39.659456 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:39.659460 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:39.659462 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:39.659487 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:39.660821 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:39.661515 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:39.661518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:39.661519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:39.661606 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:39.661907 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:39.661909 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:39.661910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:39.662093 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:39.662165 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:39.662688 (+ 523us) server_negotiation.cc:299] Negotiation successful
0923 17:28:39.662815 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260923 17:28:39.663882 29718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:39.649860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:40791 (local address 127.11.229.3:60531)
0923 17:28:39.650606 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:39.650662 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:39.651479 (+ 817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:39.652019 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:39.652026 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:39.652403 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:39.652919 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:39.652933 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:39.653905 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:39.653908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:39.655009 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:39.655028 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:39.656122 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:39.656126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:39.656378 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:39.657230 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:39.657260 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:39.659265 (+ 2005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:39.660954 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:39.660959 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:39.660977 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:39.661422 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:39.661693 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:39.661696 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:39.661697 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:39.661843 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:39.662173 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:39.662179 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:39.662445 (+ 266us) client_negotiation.cc:769] Sending connection context
0923 17:28:39.662609 (+ 164us) client_negotiation.cc:240] Negotiation successful
0923 17:28:39.662907 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":131,"threads_started":1}
I20260923 17:28:39.665218 29716 heartbeater.cc:347] Connected to a master server at 127.11.229.62:40791
I20260923 17:28:39.665573 29716 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:39.666116 29716 heartbeater.cc:511] Master 127.11.229.62:40791 requested a full tablet report, sending...
I20260923 17:28:39.667258 29257 ts_manager.cc:194] Registered new tserver with Master: 127514e6d1c54c60995e114316b8ad1a (127.11.229.3:33029)
I20260923 17:28:39.667865 29257 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.3@KRBTEST.COM'} at 127.11.229.3:60531
I20260923 17:28:39.670949 12180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:28:39.680094 29296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:39.672973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55196 (local address 127.11.229.62:40791)
0923 17:28:39.673114 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:39.673118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:39.673330 (+ 212us) server_negotiation.cc:407] Connection header received
0923 17:28:39.673429 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:39.673432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:39.673472 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:39.673548 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:39.674220 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:39.674740 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:39.675561 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:39.675762 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:39.675837 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:39.677232 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:39.677234 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:39.677236 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:39.677255 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:39.678587 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:39.678949 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:39.678951 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:39.678953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:39.679024 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:39.679283 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:39.679286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:39.679288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:39.679522 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:39.679635 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:39.679873 (+ 238us) server_negotiation.cc:299] Negotiation successful
0923 17:28:39.679967 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 17:28:39.683075 29257 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:28:39.686246 29257 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:28:39.702775 29727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:39.697115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:33029 (local address 127.0.0.1:39110)
0923 17:28:39.697976 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:39.698014 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:39.698161 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:39.698669 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:39.698676 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:39.698698 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:39.698852 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:39.698860 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:39.700145 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:39.700148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:39.701230 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:39.701240 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:39.702149 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:39.702152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:39.702299 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:39.702347 (+ 48us) client_negotiation.cc:769] Sending connection context
0923 17:28:39.702462 (+ 115us) client_negotiation.cc:240] Negotiation successful
0923 17:28:39.702585 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":21,"spinlock_wait_cycles":12672,"thread_start_us":152,"threads_started":1}
I20260923 17:28:39.702773 29728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:39.697222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39110 (local address 127.11.229.3:33029)
0923 17:28:39.697710 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:39.697722 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:39.698109 (+ 387us) server_negotiation.cc:407] Connection header received
0923 17:28:39.698323 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:39.698327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:39.698498 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:39.698673 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:39.698981 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:39.699990 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:39.701419 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:39.702032 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:39.702147 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:39.702303 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:39.702515 (+ 212us) server_negotiation.cc:299] Negotiation successful
0923 17:28:39.702618 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":168,"threads_started":1}
I20260923 17:28:39.703649 29730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:39.697193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:40991 (local address 127.0.0.1:39376)
0923 17:28:39.697989 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:39.698032 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:39.698161 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:39.699277 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:39.699280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:39.699297 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:39.699394 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:39.699401 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:39.700739 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:39.700743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:39.701968 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:39.701981 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:39.703250 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:39.703254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:39.703357 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:39.703383 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 17:28:39.703449 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 17:28:39.703521 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"thread_start_us":645,"threads_started":1}
I20260923 17:28:39.704216 29729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:39.697521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39376 (local address 127.11.229.2:40991)
0923 17:28:39.698880 (+ 1359us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:39.698893 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:39.698917 (+ 24us) server_negotiation.cc:407] Connection header received
0923 17:28:39.698994 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:39.698999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:39.699153 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:39.699298 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:39.699527 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:39.700599 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:39.702075 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:39.702905 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:39.703023 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:39.703170 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:39.703947 (+ 777us) server_negotiation.cc:299] Negotiation successful
0923 17:28:39.704063 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1259,"thread_start_us":100,"threads_started":1}
I20260923 17:28:39.704764 29731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:39.698464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:38321 (local address 127.0.0.1:56832)
0923 17:28:39.699534 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:39.699547 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:39.699653 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:39.700309 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:39.700311 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:39.700327 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:39.700471 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:39.700480 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:39.701984 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:39.701989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:39.703095 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:39.703106 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:39.704345 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:39.704350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:39.704446 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:39.704468 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 17:28:39.704565 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 17:28:39.704636 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1007,"mutex_wait_us":35,"thread_start_us":88,"threads_started":1}
I20260923 17:28:39.705029 29732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:39.698650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56832 (local address 127.11.229.1:38321)
0923 17:28:39.699018 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:39.699026 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:39.699639 (+ 613us) server_negotiation.cc:407] Connection header received
0923 17:28:39.699813 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:39.699819 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:39.700011 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:39.700151 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:39.700606 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:39.701859 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:39.703245 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:39.704027 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:39.704133 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:39.704272 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:39.704602 (+ 330us) server_negotiation.cc:299] Negotiation successful
0923 17:28:39.704702 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":73,"threads_started":1}
I20260923 17:28:39.707031 29650 tablet_service.cc:1467] Processing CreateTablet for tablet 5775c3f65dd944c8b441d69a8c5df0c8 (DEFAULT_TABLE table=test-table [id=b2db58a1a40b4d47905e5dc8293871c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:39.707168 29516 tablet_service.cc:1467] Processing CreateTablet for tablet 5775c3f65dd944c8b441d69a8c5df0c8 (DEFAULT_TABLE table=test-table [id=b2db58a1a40b4d47905e5dc8293871c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:39.708305 29516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5775c3f65dd944c8b441d69a8c5df0c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:39.708315 29650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5775c3f65dd944c8b441d69a8c5df0c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:39.708055 29382 tablet_service.cc:1467] Processing CreateTablet for tablet 5775c3f65dd944c8b441d69a8c5df0c8 (DEFAULT_TABLE table=test-table [id=b2db58a1a40b4d47905e5dc8293871c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:39.709049 29382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5775c3f65dd944c8b441d69a8c5df0c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:39.715143 29733 tablet_bootstrap.cc:492] T 5775c3f65dd944c8b441d69a8c5df0c8 P 127514e6d1c54c60995e114316b8ad1a: Bootstrap starting.
I20260923 17:28:39.716617 29734 tablet_bootstrap.cc:492] T 5775c3f65dd944c8b441d69a8c5df0c8 P 5930b16495e748388349463fe3b22f80: Bootstrap starting.
I20260923 17:28:39.717509 29733 tablet_bootstrap.cc:654] T 5775c3f65dd944c8b441d69a8c5df0c8 P 127514e6d1c54c60995e114316b8ad1a: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:39.717614 29735 tablet_bootstrap.cc:492] T 5775c3f65dd944c8b441d69a8c5df0c8 P 9768fd85be8e4efbb467ec3635422757: Bootstrap starting.
I20260923 17:28:39.718881 29733 log.cc:824] T 5775c3f65dd944c8b441d69a8c5df0c8 P 127514e6d1c54c60995e114316b8ad1a: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:39.719144 29734 tablet_bootstrap.cc:654] T 5775c3f65dd944c8b441d69a8c5df0c8 P 5930b16495e748388349463fe3b22f80: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:39.719751 29735 tablet_bootstrap.cc:654] T 5775c3f65dd944c8b441d69a8c5df0c8 P 9768fd85be8e4efbb467ec3635422757: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:39.719997 29734 log.cc:824] T 5775c3f65dd944c8b441d69a8c5df0c8 P 5930b16495e748388349463fe3b22f80: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:39.720559 29735 log.cc:824] T 5775c3f65dd944c8b441d69a8c5df0c8 P 9768fd85be8e4efbb467ec3635422757: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:39.721131 29733 tablet_bootstrap.cc:492] T 5775c3f65dd944c8b441d69a8c5df0c8 P 127514e6d1c54c60995e114316b8ad1a: No bootstrap required, opened a new log
I20260923 17:28:39.721395 29733 ts_tablet_manager.cc:1397] T 5775c3f65dd944c8b441d69a8c5df0c8 P 127514e6d1c54c60995e114316b8ad1a: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260923 17:28:39.721988 29734 tablet_bootstrap.cc:492] T 5775c3f65dd944c8b441d69a8c5df0c8 P 5930b16495e748388349463fe3b22f80: No bootstrap required, opened a new log
I20260923 17:28:39.722184 29734 ts_tablet_manager.cc:1397] T 5775c3f65dd944c8b441d69a8c5df0c8 P 5930b16495e748388349463fe3b22f80: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260923 17:28:39.722602 29735 tablet_bootstrap.cc:492] T 5775c3f65dd944c8b441d69a8c5df0c8 P 9768fd85be8e4efbb467ec3635422757: No bootstrap required, opened a new log
I20260923 17:28:39.722795 29735 ts_tablet_manager.cc:1397] T 5775c3f65dd944c8b441d69a8c5df0c8 P 9768fd85be8e4efbb467ec3635422757: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 17:28:39.725234 29734 raft_consensus.cc:348] T 5775c3f65dd944c8b441d69a8c5df0c8 P 5930b16495e748388349463fe3b22f80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "127514e6d1c54c60995e114316b8ad1a" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33029 } } peers { permanent_uuid: "5930b16495e748388349463fe3b22f80" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 40991 } } peers { permanent_uuid: "9768fd85be8e4efbb467ec3635422757" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 38321 } }
I20260923 17:28:39.725572 29734 raft_consensus.cc:374] T 5775c3f65dd944c8b441d69a8c5df0c8 P 5930b16495e748388349463fe3b22f80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:39.725628 29734 raft_consensus.cc:729] T 5775c3f65dd944c8b441d69a8c5df0c8 P 5930b16495e748388349463fe3b22f80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5930b16495e748388349463fe3b22f80, State: Initialized, Role: FOLLOWER
I20260923 17:28:39.725587 29733 raft_consensus.cc:348] T 5775c3f65dd944c8b441d69a8c5df0c8 P 127514e6d1c54c60995e114316b8ad1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "127514e6d1c54c60995e114316b8ad1a" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33029 } } peers { permanent_uuid: "5930b16495e748388349463fe3b22f80" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 40991 } } peers { permanent_uuid: "9768fd85be8e4efbb467ec3635422757" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 38321 } }
I20260923 17:28:39.725845 29733 raft_consensus.cc:374] T 5775c3f65dd944c8b441d69a8c5df0c8 P 127514e6d1c54c60995e114316b8ad1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:39.725901 29733 raft_consensus.cc:729] T 5775c3f65dd944c8b441d69a8c5df0c8 P 127514e6d1c54c60995e114316b8ad1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 127514e6d1c54c60995e114316b8ad1a, State: Initialized, Role: FOLLOWER
I20260923 17:28:39.726131 29734 consensus_queue.cc:260] T 5775c3f65dd944c8b441d69a8c5df0c8 P 5930b16495e748388349463fe3b22f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "127514e6d1c54c60995e114316b8ad1a" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33029 } } peers { permanent_uuid: "5930b16495e748388349463fe3b22f80" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 40991 } } peers { permanent_uuid: "9768fd85be8e4efbb467ec3635422757" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 38321 } }
I20260923 17:28:39.726347 29733 consensus_queue.cc:260] T 5775c3f65dd944c8b441d69a8c5df0c8 P 127514e6d1c54c60995e114316b8ad1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "127514e6d1c54c60995e114316b8ad1a" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33029 } } peers { permanent_uuid: "5930b16495e748388349463fe3b22f80" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 40991 } } peers { permanent_uuid: "9768fd85be8e4efbb467ec3635422757" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 38321 } }
I20260923 17:28:39.726899 29582 heartbeater.cc:503] Master 127.11.229.62:40791 was elected leader, sending a full tablet report...
I20260923 17:28:39.727252 29734 ts_tablet_manager.cc:1428] T 5775c3f65dd944c8b441d69a8c5df0c8 P 5930b16495e748388349463fe3b22f80: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260923 17:28:39.727295 29733 ts_tablet_manager.cc:1428] T 5775c3f65dd944c8b441d69a8c5df0c8 P 127514e6d1c54c60995e114316b8ad1a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260923 17:28:39.727628 29716 heartbeater.cc:503] Master 127.11.229.62:40791 was elected leader, sending a full tablet report...
I20260923 17:28:39.727490 29735 raft_consensus.cc:348] T 5775c3f65dd944c8b441d69a8c5df0c8 P 9768fd85be8e4efbb467ec3635422757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "127514e6d1c54c60995e114316b8ad1a" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33029 } } peers { permanent_uuid: "5930b16495e748388349463fe3b22f80" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 40991 } } peers { permanent_uuid: "9768fd85be8e4efbb467ec3635422757" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 38321 } }
I20260923 17:28:39.727735 29735 raft_consensus.cc:374] T 5775c3f65dd944c8b441d69a8c5df0c8 P 9768fd85be8e4efbb467ec3635422757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:39.727797 29735 raft_consensus.cc:729] T 5775c3f65dd944c8b441d69a8c5df0c8 P 9768fd85be8e4efbb467ec3635422757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9768fd85be8e4efbb467ec3635422757, State: Initialized, Role: FOLLOWER
I20260923 17:28:39.728209 29735 consensus_queue.cc:260] T 5775c3f65dd944c8b441d69a8c5df0c8 P 9768fd85be8e4efbb467ec3635422757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "127514e6d1c54c60995e114316b8ad1a" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33029 } } peers { permanent_uuid: "5930b16495e748388349463fe3b22f80" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 40991 } } peers { permanent_uuid: "9768fd85be8e4efbb467ec3635422757" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 38321 } }
I20260923 17:28:39.729029 29448 heartbeater.cc:503] Master 127.11.229.62:40791 was elected leader, sending a full tablet report...
I20260923 17:28:39.730902 29735 ts_tablet_manager.cc:1428] T 5775c3f65dd944c8b441d69a8c5df0c8 P 9768fd85be8e4efbb467ec3635422757: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260923 17:28:39.818455 29740 raft_consensus.cc:482] T 5775c3f65dd944c8b441d69a8c5df0c8 P 127514e6d1c54c60995e114316b8ad1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:28:39.818658 29740 raft_consensus.cc:504] T 5775c3f65dd944c8b441d69a8c5df0c8 P 127514e6d1c54c60995e114316b8ad1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "127514e6d1c54c60995e114316b8ad1a" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33029 } } peers { permanent_uuid: "5930b16495e748388349463fe3b22f80" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 40991 } } peers { permanent_uuid: "9768fd85be8e4efbb467ec3635422757" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 38321 } }
I20260923 17:28:39.819772 29740 leader_election.cc:288] T 5775c3f65dd944c8b441d69a8c5df0c8 P 127514e6d1c54c60995e114316b8ad1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5930b16495e748388349463fe3b22f80 (127.11.229.2:40991), 9768fd85be8e4efbb467ec3635422757 (127.11.229.1:38321)
I20260923 17:28:39.824839 29732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:39.820147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:38083 (local address 127.11.229.1:38321)
0923 17:28:39.820276 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:39.820279 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:39.820725 (+ 446us) server_negotiation.cc:407] Connection header received
0923 17:28:39.820946 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:39.820949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:39.820999 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:39.821077 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:39.821380 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:39.822017 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:39.823674 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:39.824327 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:39.824441 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:39.824486 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:39.824658 (+ 172us) server_negotiation.cc:299] Negotiation successful
0923 17:28:39.824730 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 17:28:39.824839 29729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:39.820129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:44177 (local address 127.11.229.2:40991)
0923 17:28:39.820288 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:39.820294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:39.820310 (+ 16us) server_negotiation.cc:407] Connection header received
0923 17:28:39.820379 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:39.820384 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:39.820447 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:39.820531 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:39.820925 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:39.821731 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:39.823529 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:39.824190 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:39.824330 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:39.824411 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:39.824595 (+ 184us) server_negotiation.cc:299] Negotiation successful
0923 17:28:39.824697 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:28:39.824847 29742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:39.820003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:38321 (local address 127.11.229.3:38083)
0923 17:28:39.820620 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:39.820633 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:39.820801 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:39.821092 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:39.821094 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:39.821109 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:39.821209 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:39.821213 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:39.822124 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:39.822127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:39.823548 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:39.823555 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:39.824433 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:39.824436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:39.824531 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:39.824562 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 17:28:39.824628 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 17:28:39.824677 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":294,"thread_start_us":130,"threads_started":1}
I20260923 17:28:39.824841 29718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:39.820003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:40991 (local address 127.11.229.3:44177)
0923 17:28:39.820113 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:39.820131 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:39.820223 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:39.820568 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:39.820572 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:39.820593 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:39.820793 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:39.820801 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:39.821871 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:39.821874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:39.823356 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:39.823369 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:39.824325 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:39.824328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:39.824456 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:39.824487 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 17:28:39.824570 (+ 83us) client_negotiation.cc:240] Negotiation successful
0923 17:28:39.824662 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260923 17:28:39.825735 29536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5775c3f65dd944c8b441d69a8c5df0c8" candidate_uuid: "127514e6d1c54c60995e114316b8ad1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5930b16495e748388349463fe3b22f80" is_pre_election: true
I20260923 17:28:39.825726 29402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5775c3f65dd944c8b441d69a8c5df0c8" candidate_uuid: "127514e6d1c54c60995e114316b8ad1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9768fd85be8e4efbb467ec3635422757" is_pre_election: true
I20260923 17:28:39.826062 29536 raft_consensus.cc:2454] T 5775c3f65dd944c8b441d69a8c5df0c8 P 5930b16495e748388349463fe3b22f80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 127514e6d1c54c60995e114316b8ad1a in term 0.
I20260923 17:28:39.826063 29402 raft_consensus.cc:2454] T 5775c3f65dd944c8b441d69a8c5df0c8 P 9768fd85be8e4efbb467ec3635422757 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 127514e6d1c54c60995e114316b8ad1a in term 0.
I20260923 17:28:39.826619 29603 leader_election.cc:302] T 5775c3f65dd944c8b441d69a8c5df0c8 P 127514e6d1c54c60995e114316b8ad1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 127514e6d1c54c60995e114316b8ad1a, 5930b16495e748388349463fe3b22f80, 9768fd85be8e4efbb467ec3635422757; no voters:
I20260923 17:28:39.826942 29740 raft_consensus.cc:2781] T 5775c3f65dd944c8b441d69a8c5df0c8 P 127514e6d1c54c60995e114316b8ad1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:28:39.827020 29740 raft_consensus.cc:482] T 5775c3f65dd944c8b441d69a8c5df0c8 P 127514e6d1c54c60995e114316b8ad1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:28:39.827068 29740 raft_consensus.cc:3037] T 5775c3f65dd944c8b441d69a8c5df0c8 P 127514e6d1c54c60995e114316b8ad1a [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:39.828224 29740 raft_consensus.cc:504] T 5775c3f65dd944c8b441d69a8c5df0c8 P 127514e6d1c54c60995e114316b8ad1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "127514e6d1c54c60995e114316b8ad1a" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33029 } } peers { permanent_uuid: "5930b16495e748388349463fe3b22f80" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 40991 } } peers { permanent_uuid: "9768fd85be8e4efbb467ec3635422757" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 38321 } }
I20260923 17:28:39.828768 29740 leader_election.cc:288] T 5775c3f65dd944c8b441d69a8c5df0c8 P 127514e6d1c54c60995e114316b8ad1a [CANDIDATE]: Term 1 election: Requested vote from peers 5930b16495e748388349463fe3b22f80 (127.11.229.2:40991), 9768fd85be8e4efbb467ec3635422757 (127.11.229.1:38321)
I20260923 17:28:39.829310 29536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5775c3f65dd944c8b441d69a8c5df0c8" candidate_uuid: "127514e6d1c54c60995e114316b8ad1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5930b16495e748388349463fe3b22f80"
I20260923 17:28:39.829310 29402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5775c3f65dd944c8b441d69a8c5df0c8" candidate_uuid: "127514e6d1c54c60995e114316b8ad1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9768fd85be8e4efbb467ec3635422757"
I20260923 17:28:39.829468 29402 raft_consensus.cc:3037] T 5775c3f65dd944c8b441d69a8c5df0c8 P 9768fd85be8e4efbb467ec3635422757 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:39.829466 29536 raft_consensus.cc:3037] T 5775c3f65dd944c8b441d69a8c5df0c8 P 5930b16495e748388349463fe3b22f80 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:39.830860 29402 raft_consensus.cc:2454] T 5775c3f65dd944c8b441d69a8c5df0c8 P 9768fd85be8e4efbb467ec3635422757 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 127514e6d1c54c60995e114316b8ad1a in term 1.
I20260923 17:28:39.830869 29536 raft_consensus.cc:2454] T 5775c3f65dd944c8b441d69a8c5df0c8 P 5930b16495e748388349463fe3b22f80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 127514e6d1c54c60995e114316b8ad1a in term 1.
I20260923 17:28:39.831234 29604 leader_election.cc:302] T 5775c3f65dd944c8b441d69a8c5df0c8 P 127514e6d1c54c60995e114316b8ad1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 127514e6d1c54c60995e114316b8ad1a, 5930b16495e748388349463fe3b22f80; no voters:
I20260923 17:28:39.831439 29740 raft_consensus.cc:2781] T 5775c3f65dd944c8b441d69a8c5df0c8 P 127514e6d1c54c60995e114316b8ad1a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:39.831674 29740 raft_consensus.cc:686] T 5775c3f65dd944c8b441d69a8c5df0c8 P 127514e6d1c54c60995e114316b8ad1a [term 1 LEADER]: Becoming Leader. State: Replica: 127514e6d1c54c60995e114316b8ad1a, State: Running, Role: LEADER
I20260923 17:28:39.832032 29740 consensus_queue.cc:237] T 5775c3f65dd944c8b441d69a8c5df0c8 P 127514e6d1c54c60995e114316b8ad1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "127514e6d1c54c60995e114316b8ad1a" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33029 } } peers { permanent_uuid: "5930b16495e748388349463fe3b22f80" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 40991 } } peers { permanent_uuid: "9768fd85be8e4efbb467ec3635422757" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 38321 } }
I20260923 17:28:39.835628 29255 catalog_manager.cc:5495] T 5775c3f65dd944c8b441d69a8c5df0c8 P 127514e6d1c54c60995e114316b8ad1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 127514e6d1c54c60995e114316b8ad1a (127.11.229.3). New cstate: current_term: 1 leader_uuid: "127514e6d1c54c60995e114316b8ad1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "127514e6d1c54c60995e114316b8ad1a" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 33029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5930b16495e748388349463fe3b22f80" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 40991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9768fd85be8e4efbb467ec3635422757" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 38321 } health_report { overall_health: UNKNOWN } } }
I20260923 17:28:39.856639 29728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:39.852968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39120 (local address 127.11.229.3:33029)
0923 17:28:39.853118 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:39.853123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:39.853140 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:28:39.853323 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:39.853326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:39.853386 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:39.853500 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:28:39.853748 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:39.854263 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:39.855115 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:39.855338 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:39.855445 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:39.855765 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:28:39.856199 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:28:39.856331 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:39.856394 (+ 63us) server_negotiation.cc:299] Negotiation successful
0923 17:28:39.856454 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:28:39.862787 29536 raft_consensus.cc:1261] T 5775c3f65dd944c8b441d69a8c5df0c8 P 5930b16495e748388349463fe3b22f80 [term 1 FOLLOWER]: Refusing update from remote peer 127514e6d1c54c60995e114316b8ad1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:28:39.862785 29402 raft_consensus.cc:1261] T 5775c3f65dd944c8b441d69a8c5df0c8 P 9768fd85be8e4efbb467ec3635422757 [term 1 FOLLOWER]: Refusing update from remote peer 127514e6d1c54c60995e114316b8ad1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:28:39.863528 29740 consensus_queue.cc:1035] T 5775c3f65dd944c8b441d69a8c5df0c8 P 127514e6d1c54c60995e114316b8ad1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5930b16495e748388349463fe3b22f80" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 40991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:28:39.863734 29743 consensus_queue.cc:1035] T 5775c3f65dd944c8b441d69a8c5df0c8 P 127514e6d1c54c60995e114316b8ad1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9768fd85be8e4efbb467ec3635422757" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:28:39.873757 29747 mvcc.cc:205] Tried to move back new op lower bound from 7332595793350225920 to 7332595793234845696. Current Snapshot: MvccSnapshot[applied={T|T < 7332595793350225920}]
I20260923 17:28:39.882431 29255 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:28:39.882634 29255 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:28:39.885432 29255 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 21490c5f8c4f4c5fadb278de71a851d5: Sending DeleteTablet for 3 replicas of tablet 5775c3f65dd944c8b441d69a8c5df0c8
I20260923 17:28:39.886320 29650 tablet_service.cc:1514] Processing DeleteTablet for tablet 5775c3f65dd944c8b441d69a8c5df0c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:39 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:39110
I20260923 17:28:39.886332 29516 tablet_service.cc:1514] Processing DeleteTablet for tablet 5775c3f65dd944c8b441d69a8c5df0c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:39 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:39376
I20260923 17:28:39.886464 29382 tablet_service.cc:1514] Processing DeleteTablet for tablet 5775c3f65dd944c8b441d69a8c5df0c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:39 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:56832
I20260923 17:28:39.886929 29757 tablet_replica.cc:330] T 5775c3f65dd944c8b441d69a8c5df0c8 P 5930b16495e748388349463fe3b22f80: stopping tablet replica
I20260923 17:28:39.887204 29758 tablet_replica.cc:330] T 5775c3f65dd944c8b441d69a8c5df0c8 P 9768fd85be8e4efbb467ec3635422757: stopping tablet replica
I20260923 17:28:39.887315 29757 raft_consensus.cc:2229] T 5775c3f65dd944c8b441d69a8c5df0c8 P 5930b16495e748388349463fe3b22f80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:28:39.887490 29758 raft_consensus.cc:2229] T 5775c3f65dd944c8b441d69a8c5df0c8 P 9768fd85be8e4efbb467ec3635422757 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:28:39.887676 29757 raft_consensus.cc:2258] T 5775c3f65dd944c8b441d69a8c5df0c8 P 5930b16495e748388349463fe3b22f80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:39.887730 29758 raft_consensus.cc:2258] T 5775c3f65dd944c8b441d69a8c5df0c8 P 9768fd85be8e4efbb467ec3635422757 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:39.887938 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 29319
I20260923 17:28:39.889474 29757 ts_tablet_manager.cc:1905] T 5775c3f65dd944c8b441d69a8c5df0c8 P 5930b16495e748388349463fe3b22f80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:39.893417 29757 ts_tablet_manager.cc:1918] T 5775c3f65dd944c8b441d69a8c5df0c8 P 5930b16495e748388349463fe3b22f80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:28:39.893546 29757 log.cc:1196] T 5775c3f65dd944c8b441d69a8c5df0c8 P 5930b16495e748388349463fe3b22f80: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184271597484-12180-0/minicluster-data/ts-1/wal/wals/5775c3f65dd944c8b441d69a8c5df0c8
I20260923 17:28:39.893749 29756 tablet_replica.cc:330] T 5775c3f65dd944c8b441d69a8c5df0c8 P 127514e6d1c54c60995e114316b8ad1a: stopping tablet replica
I20260923 17:28:39.893926 29757 ts_tablet_manager.cc:1939] T 5775c3f65dd944c8b441d69a8c5df0c8 P 5930b16495e748388349463fe3b22f80: Deleting consensus metadata
I20260923 17:28:39.894269 29756 raft_consensus.cc:2229] T 5775c3f65dd944c8b441d69a8c5df0c8 P 127514e6d1c54c60995e114316b8ad1a [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:28:39.894735 29756 raft_consensus.cc:2258] T 5775c3f65dd944c8b441d69a8c5df0c8 P 127514e6d1c54c60995e114316b8ad1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:39.895488 29242 catalog_manager.cc:4841] TS 5930b16495e748388349463fe3b22f80 (127.11.229.2:40991): tablet 5775c3f65dd944c8b441d69a8c5df0c8 (table test-table [id=b2db58a1a40b4d47905e5dc8293871c6]) successfully deleted
I20260923 17:28:39.896701 29756 ts_tablet_manager.cc:1905] T 5775c3f65dd944c8b441d69a8c5df0c8 P 127514e6d1c54c60995e114316b8ad1a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260923 17:28:39.897307 29241 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.229.1:38321 (error 108)
I20260923 17:28:39.897482 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 29453
I20260923 17:28:39.899396 29731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:39.898678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:38321 (local address 127.0.0.1:56838)
0923 17:28:39.899208 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:39.899298 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.1:38321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426}
W20260923 17:28:39.900559 29241 catalog_manager.cc:4584] TS 9768fd85be8e4efbb467ec3635422757 (127.11.229.1:38321): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5775c3f65dd944c8b441d69a8c5df0c8: Network error: Client connection negotiation failed: client connection to 127.11.229.1:38321: connect: Connection refused (error 111)
I20260923 17:28:39.904544 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 29587
I20260923 17:28:39.910316 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 29225
2026-09-23T17:28:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2448 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-23T17:28:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:28:39Z Disabled control of system clock
I20260923 17:28:39.943683 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:44117
--webserver_interface=127.11.229.62
--webserver_port=0
--builtin_ntp_servers=127.11.229.20:35973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:44117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 17:28:40.066527 29763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:40.066859 29763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:40.066900 29763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:40.071209 29763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:28:40.071295 29763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:40.071310 29763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:40.071328 29763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:28:40.071343 29763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:28:40.076416 29763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:35973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:44117
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:44117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:40.077622 29763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:40.079030 29763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:40.088127 29771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:40.088164 29769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:40.088188 29768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:40.088358 29763 server_base.cc:1034] running on GCE node
I20260923 17:28:40.089062 29763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:40.090329 29763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:40.091518 29763 hybrid_clock.cc:648] HybridClock initialized: now 1790184520091496 us; error 38 us; skew 500 ppm
I20260923 17:28:40.093703 29763 webserver.cc:460] Webserver started at http://127.11.229.62:40519/ using document root <none> and password file <none>
I20260923 17:28:40.094326 29763 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:40.094391 29763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:40.094620 29763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:40.096802 29763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "a43992801c524469933c4dc4e0978d70"
format_stamp: "Formatted at 2026-09-23 17:28:40 on dist-test-slave-ptl9"
server_key: "f42bdf7cd61e27cc1f880b945421e4ca"
server_key_iv: "b7f8d030c311ca2dfa3397d6fd843f28"
server_key_version: "encryptionkey@0"
I20260923 17:28:40.097455 29763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "a43992801c524469933c4dc4e0978d70"
format_stamp: "Formatted at 2026-09-23 17:28:40 on dist-test-slave-ptl9"
server_key: "f42bdf7cd61e27cc1f880b945421e4ca"
server_key_iv: "b7f8d030c311ca2dfa3397d6fd843f28"
server_key_version: "encryptionkey@0"
I20260923 17:28:40.101588 29763 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:28:40.104341 29777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:40.105904 29763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:28:40.106068 29763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "a43992801c524469933c4dc4e0978d70"
format_stamp: "Formatted at 2026-09-23 17:28:40 on dist-test-slave-ptl9"
server_key: "f42bdf7cd61e27cc1f880b945421e4ca"
server_key_iv: "b7f8d030c311ca2dfa3397d6fd843f28"
server_key_version: "encryptionkey@0"
I20260923 17:28:40.106215 29763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:40.117599 29763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:40.118415 29763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:40.118637 29763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:40.127063 29763 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:44117
I20260923 17:28:40.127089 29829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:44117 every 8 connection(s)
I20260923 17:28:40.128208 29763 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:28:40.130788 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 29763
I20260923 17:28:40.131033 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:28:40.131354 12180 external_mini_cluster.cc:1428] Setting key de01f556fc340de635a221be7e0bcee0
I20260923 17:28:40.131860 29830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:40.139307 29830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a43992801c524469933c4dc4e0978d70: Bootstrap starting.
I20260923 17:28:40.142007 29830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a43992801c524469933c4dc4e0978d70: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:40.142803 29833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:40.132875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55452 (local address 127.11.229.62:44117)
0923 17:28:40.133685 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:40.133703 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:40.133741 (+ 38us) server_negotiation.cc:407] Connection header received
0923 17:28:40.135176 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:40.135199 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:40.135655 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:40.136038 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:40.136507 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:40.137581 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:40.138522 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:40.139013 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:40.139188 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:40.140465 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:40.140474 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:40.140499 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:40.141095 (+ 596us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:40.141185 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:40.141212 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:40.141389 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:40.141626 (+ 237us) server_negotiation.cc:299] Negotiation successful
0923 17:28:40.141958 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":169,"threads_started":1}
I20260923 17:28:40.142815 29830 log.cc:824] T 00000000000000000000000000000000 P a43992801c524469933c4dc4e0978d70: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:40.145118 29830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a43992801c524469933c4dc4e0978d70: No bootstrap required, opened a new log
I20260923 17:28:40.149101 29830 raft_consensus.cc:348] T 00000000000000000000000000000000 P a43992801c524469933c4dc4e0978d70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43992801c524469933c4dc4e0978d70" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 44117 } }
I20260923 17:28:40.149437 29830 raft_consensus.cc:374] T 00000000000000000000000000000000 P a43992801c524469933c4dc4e0978d70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:40.149518 29830 raft_consensus.cc:729] T 00000000000000000000000000000000 P a43992801c524469933c4dc4e0978d70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a43992801c524469933c4dc4e0978d70, State: Initialized, Role: FOLLOWER
I20260923 17:28:40.149976 29830 consensus_queue.cc:260] T 00000000000000000000000000000000 P a43992801c524469933c4dc4e0978d70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43992801c524469933c4dc4e0978d70" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 44117 } }
I20260923 17:28:40.150151 29830 raft_consensus.cc:388] T 00000000000000000000000000000000 P a43992801c524469933c4dc4e0978d70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:28:40.150241 29830 raft_consensus.cc:482] T 00000000000000000000000000000000 P a43992801c524469933c4dc4e0978d70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:28:40.150321 29830 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a43992801c524469933c4dc4e0978d70 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:40.151343 29830 raft_consensus.cc:504] T 00000000000000000000000000000000 P a43992801c524469933c4dc4e0978d70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43992801c524469933c4dc4e0978d70" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 44117 } }
I20260923 17:28:40.151672 29830 leader_election.cc:302] T 00000000000000000000000000000000 P a43992801c524469933c4dc4e0978d70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a43992801c524469933c4dc4e0978d70; no voters:
I20260923 17:28:40.151970 29830 leader_election.cc:288] T 00000000000000000000000000000000 P a43992801c524469933c4dc4e0978d70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:28:40.152117 29835 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a43992801c524469933c4dc4e0978d70 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:40.152418 29835 raft_consensus.cc:686] T 00000000000000000000000000000000 P a43992801c524469933c4dc4e0978d70 [term 1 LEADER]: Becoming Leader. State: Replica: a43992801c524469933c4dc4e0978d70, State: Running, Role: LEADER
I20260923 17:28:40.152735 29835 consensus_queue.cc:237] T 00000000000000000000000000000000 P a43992801c524469933c4dc4e0978d70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43992801c524469933c4dc4e0978d70" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 44117 } }
I20260923 17:28:40.153388 29830 sys_catalog.cc:564] T 00000000000000000000000000000000 P a43992801c524469933c4dc4e0978d70 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:28:40.154611 29837 sys_catalog.cc:455] T 00000000000000000000000000000000 P a43992801c524469933c4dc4e0978d70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a43992801c524469933c4dc4e0978d70. Latest consensus state: current_term: 1 leader_uuid: "a43992801c524469933c4dc4e0978d70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43992801c524469933c4dc4e0978d70" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 44117 } } }
I20260923 17:28:40.154611 29836 sys_catalog.cc:455] T 00000000000000000000000000000000 P a43992801c524469933c4dc4e0978d70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a43992801c524469933c4dc4e0978d70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43992801c524469933c4dc4e0978d70" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 44117 } } }
I20260923 17:28:40.154778 29836 sys_catalog.cc:458] T 00000000000000000000000000000000 P a43992801c524469933c4dc4e0978d70 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:40.154778 29837 sys_catalog.cc:458] T 00000000000000000000000000000000 P a43992801c524469933c4dc4e0978d70 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:40.155268 29844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:28:40.158524 29844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:28:40.165109 29844 catalog_manager.cc:1347] Generated new cluster ID: 9078a293e53b41ab89bb71fc21dcbddb
I20260923 17:28:40.165198 29844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:28:40.174934 29844 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:28:40.175972 29844 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:28:40.181232 29844 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a43992801c524469933c4dc4e0978d70: Generated new TSK 0
I20260923 17:28:40.182175 29844 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:28:40.194202 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.1:0
--local_ip_for_outbound_sockets=127.11.229.1
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:44117
--builtin_ntp_servers=127.11.229.20:35973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 17:28:40.320545 29854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:40.320842 29854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:40.320886 29854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:40.325225 29854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:40.325323 29854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:40.325392 29854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.1
I20260923 17:28:40.330565 29854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:35973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:44117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.1
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:40.331774 29854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:40.333225 29854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:40.344254 29860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:40.344255 29859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:40.344277 29862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:40.344380 29854 server_base.cc:1034] running on GCE node
I20260923 17:28:40.345124 29854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:40.345913 29854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:40.347091 29854 hybrid_clock.cc:648] HybridClock initialized: now 1790184520347075 us; error 29 us; skew 500 ppm
I20260923 17:28:40.349327 29854 webserver.cc:460] Webserver started at http://127.11.229.1:44425/ using document root <none> and password file <none>
I20260923 17:28:40.350441 29854 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:40.350744 29854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:40.351065 29854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:40.353468 29854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-0/data/instance:
uuid: "1945e6406c1146cfa755bcfef3d842b8"
format_stamp: "Formatted at 2026-09-23 17:28:40 on dist-test-slave-ptl9"
server_key: "5a1bf98f4837f2a33627ac47667fdd13"
server_key_iv: "635d5610a98fba7086863576b00b6b47"
server_key_version: "encryptionkey@0"
I20260923 17:28:40.354104 29854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance:
uuid: "1945e6406c1146cfa755bcfef3d842b8"
format_stamp: "Formatted at 2026-09-23 17:28:40 on dist-test-slave-ptl9"
server_key: "5a1bf98f4837f2a33627ac47667fdd13"
server_key_iv: "635d5610a98fba7086863576b00b6b47"
server_key_version: "encryptionkey@0"
I20260923 17:28:40.358280 29854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:28:40.361543 29868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:40.363525 29854 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:28:40.363718 29854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-0/wal
uuid: "1945e6406c1146cfa755bcfef3d842b8"
format_stamp: "Formatted at 2026-09-23 17:28:40 on dist-test-slave-ptl9"
server_key: "5a1bf98f4837f2a33627ac47667fdd13"
server_key_iv: "635d5610a98fba7086863576b00b6b47"
server_key_version: "encryptionkey@0"
I20260923 17:28:40.363874 29854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:40.390339 29854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:40.391150 29854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:40.391371 29854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:40.392040 29854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:40.393069 29854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:40.393138 29854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:40.393226 29854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:40.393287 29854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:40.403129 29854 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.1:45107
I20260923 17:28:40.403148 29981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.1:45107 every 8 connection(s)
I20260923 17:28:40.404294 29854 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
I20260923 17:28:40.412179 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 29854
I20260923 17:28:40.412338 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance
I20260923 17:28:40.412678 12180 external_mini_cluster.cc:1428] Setting key 7031d3a5621dd8891c0d866d4c55f739
I20260923 17:28:40.415824 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.2:0
--local_ip_for_outbound_sockets=127.11.229.2
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:44117
--builtin_ntp_servers=127.11.229.20:35973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 17:28:40.418871 29833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:40.407249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:33371 (local address 127.11.229.62:44117)
0923 17:28:40.407442 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:40.407449 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:40.408846 (+ 1397us) server_negotiation.cc:407] Connection header received
0923 17:28:40.411439 (+ 2593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:40.411443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:40.411516 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:40.411752 (+ 236us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:40.413609 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:40.414192 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:40.415945 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:40.416350 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:40.416574 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:40.418228 (+ 1654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:40.418233 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:40.418235 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:40.418251 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:40.418277 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:40.418287 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:40.418372 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:40.418620 (+ 248us) server_negotiation.cc:299] Negotiation successful
0923 17:28:40.418714 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 17:28:40.420140 29984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:40.407517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:44117 (local address 127.11.229.1:33371)
0923 17:28:40.408554 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:40.408625 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:40.411065 (+ 2440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:40.412200 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:40.412217 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:40.412418 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:40.413348 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:40.413379 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:40.414333 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:40.414339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:40.415673 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:40.415689 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:40.416633 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:40.416641 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:40.416910 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:40.417807 (+ 897us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:28:40.417809 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:40.417883 (+ 74us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:28:40.417974 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:40.418382 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:40.418393 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:40.418402 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 17:28:40.418552 (+ 150us) client_negotiation.cc:240] Negotiation successful
0923 17:28:40.418940 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":137,"threads_started":1}
I20260923 17:28:40.421996 29982 heartbeater.cc:347] Connected to a master server at 127.11.229.62:44117
I20260923 17:28:40.422331 29982 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:40.423144 29982 heartbeater.cc:511] Master 127.11.229.62:44117 requested a full tablet report, sending...
I20260923 17:28:40.425212 29794 ts_manager.cc:194] Registered new tserver with Master: 1945e6406c1146cfa755bcfef3d842b8 (127.11.229.1:45107)
I20260923 17:28:40.426541 29794 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.229.1:33371
W20260923 17:28:40.550292 29985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:40.550602 29985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:40.550683 29985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:40.555011 29985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:40.555080 29985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:40.555178 29985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.2
I20260923 17:28:40.560277 29985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:35973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:44117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.2
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:40.561525 29985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:40.563010 29985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:40.574571 29991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:40.574741 29985 server_base.cc:1034] running on GCE node
W20260923 17:28:40.574556 29990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:40.574556 29993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:40.575347 29985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:40.576078 29985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:40.577313 29985 hybrid_clock.cc:648] HybridClock initialized: now 1790184520577249 us; error 32 us; skew 500 ppm
I20260923 17:28:40.579447 29985 webserver.cc:460] Webserver started at http://127.11.229.2:36535/ using document root <none> and password file <none>
I20260923 17:28:40.580068 29985 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:40.580133 29985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:40.580307 29985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:40.582538 29985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-1/data/instance:
uuid: "4068c3a7651e40ae966c573380f51287"
format_stamp: "Formatted at 2026-09-23 17:28:40 on dist-test-slave-ptl9"
server_key: "6d47becc6aabb2449e09c0e4bdd0cf28"
server_key_iv: "82fb392b14c542499c5c7ae2e575c994"
server_key_version: "encryptionkey@0"
I20260923 17:28:40.583076 29985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance:
uuid: "4068c3a7651e40ae966c573380f51287"
format_stamp: "Formatted at 2026-09-23 17:28:40 on dist-test-slave-ptl9"
server_key: "6d47becc6aabb2449e09c0e4bdd0cf28"
server_key_iv: "82fb392b14c542499c5c7ae2e575c994"
server_key_version: "encryptionkey@0"
I20260923 17:28:40.587028 29985 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:28:40.589752 29999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:40.591122 29985 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:28:40.591301 29985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-1/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-1/wal
uuid: "4068c3a7651e40ae966c573380f51287"
format_stamp: "Formatted at 2026-09-23 17:28:40 on dist-test-slave-ptl9"
server_key: "6d47becc6aabb2449e09c0e4bdd0cf28"
server_key_iv: "82fb392b14c542499c5c7ae2e575c994"
server_key_version: "encryptionkey@0"
I20260923 17:28:40.591439 29985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:40.606613 29985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:40.607353 29985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:40.607549 29985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:40.608166 29985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:40.609150 29985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:40.609206 29985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:40.609292 29985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:40.609344 29985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:40.619489 29985 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.2:40391
I20260923 17:28:40.619508 30112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.2:40391 every 8 connection(s)
I20260923 17:28:40.620491 29985 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
I20260923 17:28:40.624611 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 29985
I20260923 17:28:40.624791 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance
I20260923 17:28:40.625089 12180 external_mini_cluster.cc:1428] Setting key 476d94e64081986eb423eace97fae502
I20260923 17:28:40.627187 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.3:0
--local_ip_for_outbound_sockets=127.11.229.3
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:44117
--builtin_ntp_servers=127.11.229.20:35973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 17:28:40.631076 29833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:40.622894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:57083 (local address 127.11.229.62:44117)
0923 17:28:40.623158 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:40.623167 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:40.624079 (+ 912us) server_negotiation.cc:407] Connection header received
0923 17:28:40.625943 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:40.625947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:40.626006 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:40.626112 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:40.627172 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:40.627801 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:40.628974 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:40.629199 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:40.629375 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:40.630558 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:40.630562 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:40.630565 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:40.630580 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:40.630605 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:40.630613 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:40.630687 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:40.630854 (+ 167us) server_negotiation.cc:299] Negotiation successful
0923 17:28:40.630920 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":71}
I20260923 17:28:40.632031 30115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:40.623109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:44117 (local address 127.11.229.2:57083)
0923 17:28:40.623895 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:40.623940 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:40.625698 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:40.626319 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:40.626327 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:40.626420 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:40.627033 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:40.627047 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:40.627930 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:40.627933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:40.628845 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:40.628854 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:40.629428 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:40.629433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:40.629639 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:40.630282 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:28:40.630283 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:40.630326 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:28:40.630399 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:40.630699 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:40.630710 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:40.630721 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 17:28:40.630853 (+ 132us) client_negotiation.cc:240] Negotiation successful
0923 17:28:40.631165 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":122,"threads_started":1}
I20260923 17:28:40.633473 30113 heartbeater.cc:347] Connected to a master server at 127.11.229.62:44117
I20260923 17:28:40.633764 30113 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:40.634374 30113 heartbeater.cc:511] Master 127.11.229.62:44117 requested a full tablet report, sending...
I20260923 17:28:40.635628 29794 ts_manager.cc:194] Registered new tserver with Master: 4068c3a7651e40ae966c573380f51287 (127.11.229.2:40391)
I20260923 17:28:40.636575 29794 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.229.2:57083
W20260923 17:28:40.748982 30116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:40.749315 30116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:40.749370 30116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:40.754189 30116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:40.754261 30116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:40.754318 30116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.3
I20260923 17:28:40.759719 30116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:35973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:44117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.3
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:40.761494 30116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:40.763058 30116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:40.773638 30121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:40.773733 30116 server_base.cc:1034] running on GCE node
W20260923 17:28:40.773658 30122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:40.773653 30124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:40.774379 30116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:40.775028 30116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:40.776209 30116 hybrid_clock.cc:648] HybridClock initialized: now 1790184520776183 us; error 38 us; skew 500 ppm
I20260923 17:28:40.778353 30116 webserver.cc:460] Webserver started at http://127.11.229.3:36713/ using document root <none> and password file <none>
I20260923 17:28:40.778970 30116 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:40.779034 30116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:40.779249 30116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:40.781539 30116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-2/data/instance:
uuid: "22c4bae1f7fc4f40a2034d48a6a236e4"
format_stamp: "Formatted at 2026-09-23 17:28:40 on dist-test-slave-ptl9"
server_key: "12aec7b0d27387654a9cde4eebee595e"
server_key_iv: "b7b36fd238e30c39241f34c2a9b0bc9a"
server_key_version: "encryptionkey@0"
I20260923 17:28:40.782109 30116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance:
uuid: "22c4bae1f7fc4f40a2034d48a6a236e4"
format_stamp: "Formatted at 2026-09-23 17:28:40 on dist-test-slave-ptl9"
server_key: "12aec7b0d27387654a9cde4eebee595e"
server_key_iv: "b7b36fd238e30c39241f34c2a9b0bc9a"
server_key_version: "encryptionkey@0"
I20260923 17:28:40.787297 30116 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:28:40.790354 30130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:40.791692 30116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260923 17:28:40.791841 30116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-2/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-2/wal
uuid: "22c4bae1f7fc4f40a2034d48a6a236e4"
format_stamp: "Formatted at 2026-09-23 17:28:40 on dist-test-slave-ptl9"
server_key: "12aec7b0d27387654a9cde4eebee595e"
server_key_iv: "b7b36fd238e30c39241f34c2a9b0bc9a"
server_key_version: "encryptionkey@0"
I20260923 17:28:40.791976 30116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:40.802395 30116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:40.803144 30116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:40.803354 30116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:40.803973 30116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:40.804994 30116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:40.805042 30116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:40.805109 30116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:40.805145 30116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:40.815470 30116 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.3:35131
I20260923 17:28:40.815491 30243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.3:35131 every 8 connection(s)
I20260923 17:28:40.816486 30116 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
I20260923 17:28:40.824457 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 30116
I20260923 17:28:40.824616 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance
I20260923 17:28:40.824939 12180 external_mini_cluster.cc:1428] Setting key 3884ed9af859ad4f60b6f464c1c47374
I20260923 17:28:40.829377 29833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:40.818597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:57811 (local address 127.11.229.62:44117)
0923 17:28:40.818783 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:40.818787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:40.820343 (+ 1556us) server_negotiation.cc:407] Connection header received
0923 17:28:40.822563 (+ 2220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:40.822567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:40.822647 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:40.822748 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:40.824647 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:40.825539 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:40.827213 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:40.827488 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:40.827577 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:40.828756 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:40.828759 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:40.828761 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:40.828774 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:40.828795 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:40.828803 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:40.828894 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:40.829111 (+ 217us) server_negotiation.cc:299] Negotiation successful
0923 17:28:40.829174 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260923 17:28:40.830500 30246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:40.819002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:44117 (local address 127.11.229.3:57811)
0923 17:28:40.820054 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:40.820151 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:40.822197 (+ 2046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:40.823129 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:40.823142 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:40.823287 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:40.824367 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:40.824406 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:40.825688 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:40.825692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:40.827027 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:40.827040 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:40.827631 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:40.827633 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:40.827793 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:40.828437 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:28:40.828439 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:40.828488 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:28:40.828580 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:40.828937 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:40.828946 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:40.828952 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 17:28:40.829076 (+ 124us) client_negotiation.cc:240] Negotiation successful
0923 17:28:40.829455 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":128,"threads_started":1}
I20260923 17:28:40.832109 30244 heartbeater.cc:347] Connected to a master server at 127.11.229.62:44117
I20260923 17:28:40.832554 30244 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:40.833482 30244 heartbeater.cc:511] Master 127.11.229.62:44117 requested a full tablet report, sending...
I20260923 17:28:40.835103 29794 ts_manager.cc:194] Registered new tserver with Master: 22c4bae1f7fc4f40a2034d48a6a236e4 (127.11.229.3:35131)
I20260923 17:28:40.835741 29794 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.229.3:57811
I20260923 17:28:40.839609 12180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:28:40.845512 29833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:40.841810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55468 (local address 127.11.229.62:44117)
0923 17:28:40.841950 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:40.841954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:40.842085 (+ 131us) server_negotiation.cc:407] Connection header received
0923 17:28:40.842161 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:40.842163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:40.842208 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:40.842303 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:40.842599 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:40.843118 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:40.844001 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:40.844212 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:40.844294 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:40.845062 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:40.845065 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:40.845067 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:40.845080 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:40.845098 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:40.845106 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:40.845164 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:40.845316 (+ 152us) server_negotiation.cc:299] Negotiation successful
0923 17:28:40.845378 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 17:28:40.848685 29794 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55468:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:28:40.852231 29794 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:28:40.870440 30257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:40.864327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49458 (local address 127.11.229.1:45107)
0923 17:28:40.865524 (+ 1197us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:40.865531 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:40.865558 (+ 27us) server_negotiation.cc:407] Connection header received
0923 17:28:40.865638 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:40.865643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:40.865825 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:40.865991 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:40.866312 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:40.867178 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:40.868986 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:40.869703 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:40.869853 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:40.870063 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:40.870152 (+ 89us) server_negotiation.cc:299] Negotiation successful
0923 17:28:40.870255 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1075,"thread_start_us":97,"threads_started":1}
I20260923 17:28:40.870687 30258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:40.864881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:35131 (local address 127.0.0.1:57322)
0923 17:28:40.865388 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:40.865402 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:40.865481 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:40.866325 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:40.866331 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:40.866350 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:40.866450 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:40.866457 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:40.867988 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:40.867992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:40.869237 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:40.869246 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:40.870275 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:40.870279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:40.870377 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:40.870400 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 17:28:40.870469 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 17:28:40.870544 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":94,"threads_started":1}
I20260923 17:28:40.870733 30260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:40.865314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57322 (local address 127.11.229.3:35131)
0923 17:28:40.865717 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:40.865728 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:40.865747 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:28:40.865826 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:40.865830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:40.866170 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:40.866594 (+ 424us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:40.866717 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:40.867843 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:40.869373 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:40.870150 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:40.870277 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:40.870448 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:40.870496 (+ 48us) server_negotiation.cc:299] Negotiation successful
0923 17:28:40.870577 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":157,"threads_started":1}
I20260923 17:28:40.870906 30255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:40.864088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:45107 (local address 127.0.0.1:49458)
0923 17:28:40.864981 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:40.865041 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:40.865179 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:40.865985 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:40.865991 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:40.866015 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:40.866174 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:40.866180 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:40.867342 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:40.867345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:40.868829 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:40.868841 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:40.869853 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:40.869856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:40.869964 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:40.870001 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 17:28:40.870123 (+ 122us) client_negotiation.cc:240] Negotiation successful
0923 17:28:40.870255 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":103,"threads_started":1}
I20260923 17:28:40.871462 30256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:40.864318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:40391 (local address 127.0.0.1:50950)
0923 17:28:40.865071 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:40.865085 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:40.865186 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:40.866159 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:40.866162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:40.866176 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:40.866275 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:40.866281 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:40.867920 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:40.867926 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:40.868883 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:40.868896 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:40.871027 (+ 2131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:40.871030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:40.871139 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:40.871162 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 17:28:40.871238 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 17:28:40.871334 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"mutex_wait_us":34,"thread_start_us":109,"threads_started":1}
I20260923 17:28:40.871899 30259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:40.864837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50950 (local address 127.11.229.2:40391)
0923 17:28:40.865517 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:40.865531 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:40.865559 (+ 28us) server_negotiation.cc:407] Connection header received
0923 17:28:40.865638 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:40.865643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:40.865863 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:40.866164 (+ 301us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:40.866349 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:40.867779 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:40.869011 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:40.869749 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:40.869870 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:40.870063 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:40.871638 (+ 1575us) server_negotiation.cc:299] Negotiation successful
0923 17:28:40.871738 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":114,"threads_started":1}
I20260923 17:28:40.875026 29916 tablet_service.cc:1467] Processing CreateTablet for tablet 3cdf482209a8484783087c17fc402263 (DEFAULT_TABLE table=test-table [id=3df3fd9fc1894c27bdddbbe30c53b72f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:40.875018 30176 tablet_service.cc:1467] Processing CreateTablet for tablet 3cdf482209a8484783087c17fc402263 (DEFAULT_TABLE table=test-table [id=3df3fd9fc1894c27bdddbbe30c53b72f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:40.876272 29916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cdf482209a8484783087c17fc402263. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:40.876272 30176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cdf482209a8484783087c17fc402263. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:40.877473 30047 tablet_service.cc:1467] Processing CreateTablet for tablet 3cdf482209a8484783087c17fc402263 (DEFAULT_TABLE table=test-table [id=3df3fd9fc1894c27bdddbbe30c53b72f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:40.878432 30047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cdf482209a8484783087c17fc402263. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:40.884567 30261 tablet_bootstrap.cc:492] T 3cdf482209a8484783087c17fc402263 P 4068c3a7651e40ae966c573380f51287: Bootstrap starting.
I20260923 17:28:40.885917 30262 tablet_bootstrap.cc:492] T 3cdf482209a8484783087c17fc402263 P 1945e6406c1146cfa755bcfef3d842b8: Bootstrap starting.
I20260923 17:28:40.886631 30263 tablet_bootstrap.cc:492] T 3cdf482209a8484783087c17fc402263 P 22c4bae1f7fc4f40a2034d48a6a236e4: Bootstrap starting.
I20260923 17:28:40.888093 30261 tablet_bootstrap.cc:654] T 3cdf482209a8484783087c17fc402263 P 4068c3a7651e40ae966c573380f51287: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:40.888270 30262 tablet_bootstrap.cc:654] T 3cdf482209a8484783087c17fc402263 P 1945e6406c1146cfa755bcfef3d842b8: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:40.888841 30263 tablet_bootstrap.cc:654] T 3cdf482209a8484783087c17fc402263 P 22c4bae1f7fc4f40a2034d48a6a236e4: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:40.889045 30262 log.cc:824] T 3cdf482209a8484783087c17fc402263 P 1945e6406c1146cfa755bcfef3d842b8: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:40.889046 30261 log.cc:824] T 3cdf482209a8484783087c17fc402263 P 4068c3a7651e40ae966c573380f51287: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:40.889673 30263 log.cc:824] T 3cdf482209a8484783087c17fc402263 P 22c4bae1f7fc4f40a2034d48a6a236e4: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:40.891360 30261 tablet_bootstrap.cc:492] T 3cdf482209a8484783087c17fc402263 P 4068c3a7651e40ae966c573380f51287: No bootstrap required, opened a new log
I20260923 17:28:40.891389 30262 tablet_bootstrap.cc:492] T 3cdf482209a8484783087c17fc402263 P 1945e6406c1146cfa755bcfef3d842b8: No bootstrap required, opened a new log
I20260923 17:28:40.891434 30263 tablet_bootstrap.cc:492] T 3cdf482209a8484783087c17fc402263 P 22c4bae1f7fc4f40a2034d48a6a236e4: No bootstrap required, opened a new log
I20260923 17:28:40.891597 30261 ts_tablet_manager.cc:1397] T 3cdf482209a8484783087c17fc402263 P 4068c3a7651e40ae966c573380f51287: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260923 17:28:40.891597 30262 ts_tablet_manager.cc:1397] T 3cdf482209a8484783087c17fc402263 P 1945e6406c1146cfa755bcfef3d842b8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 17:28:40.891597 30263 ts_tablet_manager.cc:1397] T 3cdf482209a8484783087c17fc402263 P 22c4bae1f7fc4f40a2034d48a6a236e4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 17:28:40.896533 30262 raft_consensus.cc:348] T 3cdf482209a8484783087c17fc402263 P 1945e6406c1146cfa755bcfef3d842b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1945e6406c1146cfa755bcfef3d842b8" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45107 } } peers { permanent_uuid: "4068c3a7651e40ae966c573380f51287" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 40391 } } peers { permanent_uuid: "22c4bae1f7fc4f40a2034d48a6a236e4" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35131 } }
I20260923 17:28:40.896473 30263 raft_consensus.cc:348] T 3cdf482209a8484783087c17fc402263 P 22c4bae1f7fc4f40a2034d48a6a236e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1945e6406c1146cfa755bcfef3d842b8" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45107 } } peers { permanent_uuid: "4068c3a7651e40ae966c573380f51287" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 40391 } } peers { permanent_uuid: "22c4bae1f7fc4f40a2034d48a6a236e4" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35131 } }
I20260923 17:28:40.896564 30261 raft_consensus.cc:348] T 3cdf482209a8484783087c17fc402263 P 4068c3a7651e40ae966c573380f51287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1945e6406c1146cfa755bcfef3d842b8" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45107 } } peers { permanent_uuid: "4068c3a7651e40ae966c573380f51287" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 40391 } } peers { permanent_uuid: "22c4bae1f7fc4f40a2034d48a6a236e4" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35131 } }
I20260923 17:28:40.896839 30262 raft_consensus.cc:374] T 3cdf482209a8484783087c17fc402263 P 1945e6406c1146cfa755bcfef3d842b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:40.896839 30263 raft_consensus.cc:374] T 3cdf482209a8484783087c17fc402263 P 22c4bae1f7fc4f40a2034d48a6a236e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:40.896888 30261 raft_consensus.cc:374] T 3cdf482209a8484783087c17fc402263 P 4068c3a7651e40ae966c573380f51287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:40.896903 30262 raft_consensus.cc:729] T 3cdf482209a8484783087c17fc402263 P 1945e6406c1146cfa755bcfef3d842b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1945e6406c1146cfa755bcfef3d842b8, State: Initialized, Role: FOLLOWER
I20260923 17:28:40.896903 30263 raft_consensus.cc:729] T 3cdf482209a8484783087c17fc402263 P 22c4bae1f7fc4f40a2034d48a6a236e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22c4bae1f7fc4f40a2034d48a6a236e4, State: Initialized, Role: FOLLOWER
I20260923 17:28:40.896950 30261 raft_consensus.cc:729] T 3cdf482209a8484783087c17fc402263 P 4068c3a7651e40ae966c573380f51287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4068c3a7651e40ae966c573380f51287, State: Initialized, Role: FOLLOWER
I20260923 17:28:40.897446 30261 consensus_queue.cc:260] T 3cdf482209a8484783087c17fc402263 P 4068c3a7651e40ae966c573380f51287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1945e6406c1146cfa755bcfef3d842b8" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45107 } } peers { permanent_uuid: "4068c3a7651e40ae966c573380f51287" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 40391 } } peers { permanent_uuid: "22c4bae1f7fc4f40a2034d48a6a236e4" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35131 } }
I20260923 17:28:40.897455 30262 consensus_queue.cc:260] T 3cdf482209a8484783087c17fc402263 P 1945e6406c1146cfa755bcfef3d842b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1945e6406c1146cfa755bcfef3d842b8" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45107 } } peers { permanent_uuid: "4068c3a7651e40ae966c573380f51287" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 40391 } } peers { permanent_uuid: "22c4bae1f7fc4f40a2034d48a6a236e4" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35131 } }
I20260923 17:28:40.897455 30263 consensus_queue.cc:260] T 3cdf482209a8484783087c17fc402263 P 22c4bae1f7fc4f40a2034d48a6a236e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1945e6406c1146cfa755bcfef3d842b8" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45107 } } peers { permanent_uuid: "4068c3a7651e40ae966c573380f51287" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 40391 } } peers { permanent_uuid: "22c4bae1f7fc4f40a2034d48a6a236e4" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35131 } }
I20260923 17:28:40.898644 30262 ts_tablet_manager.cc:1428] T 3cdf482209a8484783087c17fc402263 P 1945e6406c1146cfa755bcfef3d842b8: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260923 17:28:40.898643 30263 ts_tablet_manager.cc:1428] T 3cdf482209a8484783087c17fc402263 P 22c4bae1f7fc4f40a2034d48a6a236e4: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260923 17:28:40.898661 30113 heartbeater.cc:503] Master 127.11.229.62:44117 was elected leader, sending a full tablet report...
I20260923 17:28:40.898941 29982 heartbeater.cc:503] Master 127.11.229.62:44117 was elected leader, sending a full tablet report...
I20260923 17:28:40.898977 30244 heartbeater.cc:503] Master 127.11.229.62:44117 was elected leader, sending a full tablet report...
I20260923 17:28:40.898641 30261 ts_tablet_manager.cc:1428] T 3cdf482209a8484783087c17fc402263 P 4068c3a7651e40ae966c573380f51287: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260923 17:28:40.905484 29983 tablet.cc:2369] T 3cdf482209a8484783087c17fc402263 P 1945e6406c1146cfa755bcfef3d842b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:28:41.044044 30268 raft_consensus.cc:482] T 3cdf482209a8484783087c17fc402263 P 4068c3a7651e40ae966c573380f51287 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:28:41.044260 30268 raft_consensus.cc:504] T 3cdf482209a8484783087c17fc402263 P 4068c3a7651e40ae966c573380f51287 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1945e6406c1146cfa755bcfef3d842b8" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45107 } } peers { permanent_uuid: "4068c3a7651e40ae966c573380f51287" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 40391 } } peers { permanent_uuid: "22c4bae1f7fc4f40a2034d48a6a236e4" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35131 } }
I20260923 17:28:41.045501 30268 leader_election.cc:288] T 3cdf482209a8484783087c17fc402263 P 4068c3a7651e40ae966c573380f51287 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1945e6406c1146cfa755bcfef3d842b8 (127.11.229.1:45107), 22c4bae1f7fc4f40a2034d48a6a236e4 (127.11.229.3:35131)
I20260923 17:28:41.049124 30115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:41.045757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:45107 (local address 127.11.229.2:58493)
0923 17:28:41.045928 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:41.045947 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:41.046073 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:41.046375 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:41.046378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:41.046395 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:41.046516 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:41.046523 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:41.047257 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:41.047262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:41.048067 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:41.048076 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:41.048697 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:41.048700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:41.048800 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:41.048828 (+ 28us) client_negotiation.cc:769] Sending connection context
0923 17:28:41.048912 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 17:28:41.048986 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260923 17:28:41.049124 30257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:41.045863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:58493 (local address 127.11.229.1:45107)
0923 17:28:41.046014 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:41.046019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:41.046037 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:28:41.046197 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:41.046204 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:41.046269 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:41.046356 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:41.046629 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:41.047143 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:41.048194 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:41.048578 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:41.048701 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:41.048754 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:41.048929 (+ 175us) server_negotiation.cc:299] Negotiation successful
0923 17:28:41.048997 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 17:28:41.049714 30270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:41.046056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:35131 (local address 127.11.229.2:60343)
0923 17:28:41.046452 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:41.046463 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:41.046572 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:41.046826 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:41.046828 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:41.046844 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:41.046975 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:41.046979 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:41.047674 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:41.047678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:41.048549 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:41.048559 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:41.049297 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:41.049302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:41.049419 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:41.049440 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 17:28:41.049507 (+ 67us) client_negotiation.cc:240] Negotiation successful
0923 17:28:41.049577 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":142,"threads_started":1}
I20260923 17:28:41.049724 30260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:41.046185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:60343 (local address 127.11.229.3:35131)
0923 17:28:41.046325 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:41.046331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:41.046534 (+ 203us) server_negotiation.cc:407] Connection header received
0923 17:28:41.046690 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:41.046692 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:41.046731 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:41.046814 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:41.047076 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:41.047563 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:41.048696 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:41.049117 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:41.049246 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:41.049323 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:41.049543 (+ 220us) server_negotiation.cc:299] Negotiation successful
0923 17:28:41.049635 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 17:28:41.049912 29936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cdf482209a8484783087c17fc402263" candidate_uuid: "4068c3a7651e40ae966c573380f51287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1945e6406c1146cfa755bcfef3d842b8" is_pre_election: true
I20260923 17:28:41.050262 29936 raft_consensus.cc:2454] T 3cdf482209a8484783087c17fc402263 P 1945e6406c1146cfa755bcfef3d842b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4068c3a7651e40ae966c573380f51287 in term 0.
I20260923 17:28:41.050427 30198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cdf482209a8484783087c17fc402263" candidate_uuid: "4068c3a7651e40ae966c573380f51287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22c4bae1f7fc4f40a2034d48a6a236e4" is_pre_election: true
I20260923 17:28:41.050712 30198 raft_consensus.cc:2454] T 3cdf482209a8484783087c17fc402263 P 22c4bae1f7fc4f40a2034d48a6a236e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4068c3a7651e40ae966c573380f51287 in term 0.
I20260923 17:28:41.050765 30002 leader_election.cc:302] T 3cdf482209a8484783087c17fc402263 P 4068c3a7651e40ae966c573380f51287 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1945e6406c1146cfa755bcfef3d842b8, 4068c3a7651e40ae966c573380f51287; no voters:
I20260923 17:28:41.051048 30268 raft_consensus.cc:2781] T 3cdf482209a8484783087c17fc402263 P 4068c3a7651e40ae966c573380f51287 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:28:41.051146 30268 raft_consensus.cc:482] T 3cdf482209a8484783087c17fc402263 P 4068c3a7651e40ae966c573380f51287 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:28:41.051208 30268 raft_consensus.cc:3037] T 3cdf482209a8484783087c17fc402263 P 4068c3a7651e40ae966c573380f51287 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:41.052225 30268 raft_consensus.cc:504] T 3cdf482209a8484783087c17fc402263 P 4068c3a7651e40ae966c573380f51287 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1945e6406c1146cfa755bcfef3d842b8" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45107 } } peers { permanent_uuid: "4068c3a7651e40ae966c573380f51287" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 40391 } } peers { permanent_uuid: "22c4bae1f7fc4f40a2034d48a6a236e4" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35131 } }
I20260923 17:28:41.052601 30268 leader_election.cc:288] T 3cdf482209a8484783087c17fc402263 P 4068c3a7651e40ae966c573380f51287 [CANDIDATE]: Term 1 election: Requested vote from peers 1945e6406c1146cfa755bcfef3d842b8 (127.11.229.1:45107), 22c4bae1f7fc4f40a2034d48a6a236e4 (127.11.229.3:35131)
I20260923 17:28:41.052976 29936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cdf482209a8484783087c17fc402263" candidate_uuid: "4068c3a7651e40ae966c573380f51287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1945e6406c1146cfa755bcfef3d842b8"
I20260923 17:28:41.053020 30198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cdf482209a8484783087c17fc402263" candidate_uuid: "4068c3a7651e40ae966c573380f51287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22c4bae1f7fc4f40a2034d48a6a236e4"
I20260923 17:28:41.053109 29936 raft_consensus.cc:3037] T 3cdf482209a8484783087c17fc402263 P 1945e6406c1146cfa755bcfef3d842b8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:41.053145 30198 raft_consensus.cc:3037] T 3cdf482209a8484783087c17fc402263 P 22c4bae1f7fc4f40a2034d48a6a236e4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:41.054204 29936 raft_consensus.cc:2454] T 3cdf482209a8484783087c17fc402263 P 1945e6406c1146cfa755bcfef3d842b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4068c3a7651e40ae966c573380f51287 in term 1.
I20260923 17:28:41.054394 30198 raft_consensus.cc:2454] T 3cdf482209a8484783087c17fc402263 P 22c4bae1f7fc4f40a2034d48a6a236e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4068c3a7651e40ae966c573380f51287 in term 1.
I20260923 17:28:41.054571 30002 leader_election.cc:302] T 3cdf482209a8484783087c17fc402263 P 4068c3a7651e40ae966c573380f51287 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1945e6406c1146cfa755bcfef3d842b8, 4068c3a7651e40ae966c573380f51287; no voters:
I20260923 17:28:41.054737 30268 raft_consensus.cc:2781] T 3cdf482209a8484783087c17fc402263 P 4068c3a7651e40ae966c573380f51287 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:41.055002 30268 raft_consensus.cc:686] T 3cdf482209a8484783087c17fc402263 P 4068c3a7651e40ae966c573380f51287 [term 1 LEADER]: Becoming Leader. State: Replica: 4068c3a7651e40ae966c573380f51287, State: Running, Role: LEADER
I20260923 17:28:41.055322 30268 consensus_queue.cc:237] T 3cdf482209a8484783087c17fc402263 P 4068c3a7651e40ae966c573380f51287 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1945e6406c1146cfa755bcfef3d842b8" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45107 } } peers { permanent_uuid: "4068c3a7651e40ae966c573380f51287" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 40391 } } peers { permanent_uuid: "22c4bae1f7fc4f40a2034d48a6a236e4" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35131 } }
I20260923 17:28:41.058545 29794 catalog_manager.cc:5495] T 3cdf482209a8484783087c17fc402263 P 4068c3a7651e40ae966c573380f51287 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4068c3a7651e40ae966c573380f51287 (127.11.229.2). New cstate: current_term: 1 leader_uuid: "4068c3a7651e40ae966c573380f51287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1945e6406c1146cfa755bcfef3d842b8" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4068c3a7651e40ae966c573380f51287" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 40391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22c4bae1f7fc4f40a2034d48a6a236e4" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35131 } health_report { overall_health: UNKNOWN } } }
W20260923 17:28:41.067679 30245 tablet.cc:2369] T 3cdf482209a8484783087c17fc402263 P 22c4bae1f7fc4f40a2034d48a6a236e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:28:41.106907 30259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:41.103563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50962 (local address 127.11.229.2:40391)
0923 17:28:41.103688 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:41.103691 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:41.103707 (+ 16us) server_negotiation.cc:407] Connection header received
0923 17:28:41.103794 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:41.103796 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:41.103853 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:41.103941 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:28:41.104141 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:41.104639 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:41.105536 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:41.105749 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:41.105831 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:41.106079 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:28:41.106519 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:28:41.106612 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:41.106727 (+ 115us) server_negotiation.cc:299] Negotiation successful
0923 17:28:41.106785 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 17:28:41.113134 30198 raft_consensus.cc:1261] T 3cdf482209a8484783087c17fc402263 P 22c4bae1f7fc4f40a2034d48a6a236e4 [term 1 FOLLOWER]: Refusing update from remote peer 4068c3a7651e40ae966c573380f51287: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:28:41.113145 29936 raft_consensus.cc:1261] T 3cdf482209a8484783087c17fc402263 P 1945e6406c1146cfa755bcfef3d842b8 [term 1 FOLLOWER]: Refusing update from remote peer 4068c3a7651e40ae966c573380f51287: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:28:41.113863 30268 consensus_queue.cc:1035] T 3cdf482209a8484783087c17fc402263 P 4068c3a7651e40ae966c573380f51287 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1945e6406c1146cfa755bcfef3d842b8" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 45107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:28:41.114045 30271 consensus_queue.cc:1035] T 3cdf482209a8484783087c17fc402263 P 4068c3a7651e40ae966c573380f51287 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22c4bae1f7fc4f40a2034d48a6a236e4" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 17:28:41.121685 30114 tablet.cc:2369] T 3cdf482209a8484783087c17fc402263 P 4068c3a7651e40ae966c573380f51287: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:28:41.122874 30275 mvcc.cc:205] Tried to move back new op lower bound from 7332595798472200192 to 7332595798245289984. Current Snapshot: MvccSnapshot[applied={T|T < 7332595798472200192}]
I20260923 17:28:41.123835 30276 mvcc.cc:205] Tried to move back new op lower bound from 7332595798472200192 to 7332595798245289984. Current Snapshot: MvccSnapshot[applied={T|T < 7332595798472200192}]
I20260923 17:28:41.132117 29794 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:28:41.132314 29794 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:28:41.135270 29794 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a43992801c524469933c4dc4e0978d70: Sending DeleteTablet for 3 replicas of tablet 3cdf482209a8484783087c17fc402263
I20260923 17:28:41.136173 30047 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cdf482209a8484783087c17fc402263 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:41 UTC) from {username='slave'} at 127.0.0.1:50950
I20260923 17:28:41.136179 29916 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cdf482209a8484783087c17fc402263 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:41 UTC) from {username='slave'} at 127.0.0.1:49458
I20260923 17:28:41.137082 30284 tablet_replica.cc:330] T 3cdf482209a8484783087c17fc402263 P 4068c3a7651e40ae966c573380f51287: stopping tablet replica
I20260923 17:28:41.137240 30176 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cdf482209a8484783087c17fc402263 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:41 UTC) from {username='slave'} at 127.0.0.1:57322
I20260923 17:28:41.137485 30284 raft_consensus.cc:2229] T 3cdf482209a8484783087c17fc402263 P 4068c3a7651e40ae966c573380f51287 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:28:41.137900 30284 raft_consensus.cc:2258] T 3cdf482209a8484783087c17fc402263 P 4068c3a7651e40ae966c573380f51287 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:41.138268 30285 tablet_replica.cc:330] T 3cdf482209a8484783087c17fc402263 P 1945e6406c1146cfa755bcfef3d842b8: stopping tablet replica
I20260923 17:28:41.138573 30285 raft_consensus.cc:2229] T 3cdf482209a8484783087c17fc402263 P 1945e6406c1146cfa755bcfef3d842b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:28:41.138949 30285 raft_consensus.cc:2258] T 3cdf482209a8484783087c17fc402263 P 1945e6406c1146cfa755bcfef3d842b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:41.139597 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 29854
I20260923 17:28:41.141737 30284 ts_tablet_manager.cc:1905] T 3cdf482209a8484783087c17fc402263 P 4068c3a7651e40ae966c573380f51287: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:41.144713 30284 ts_tablet_manager.cc:1918] T 3cdf482209a8484783087c17fc402263 P 4068c3a7651e40ae966c573380f51287: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:28:41.144860 30284 log.cc:1196] T 3cdf482209a8484783087c17fc402263 P 4068c3a7651e40ae966c573380f51287: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-1/wal/wals/3cdf482209a8484783087c17fc402263
I20260923 17:28:41.145205 30284 ts_tablet_manager.cc:1939] T 3cdf482209a8484783087c17fc402263 P 4068c3a7651e40ae966c573380f51287: Deleting consensus metadata
I20260923 17:28:41.146658 29781 catalog_manager.cc:4841] TS 4068c3a7651e40ae966c573380f51287 (127.11.229.2:40391): tablet 3cdf482209a8484783087c17fc402263 (table test-table [id=3df3fd9fc1894c27bdddbbe30c53b72f]) successfully deleted
I20260923 17:28:41.149561 30286 tablet_replica.cc:330] T 3cdf482209a8484783087c17fc402263 P 22c4bae1f7fc4f40a2034d48a6a236e4: stopping tablet replica
I20260923 17:28:41.149994 30286 raft_consensus.cc:2229] T 3cdf482209a8484783087c17fc402263 P 22c4bae1f7fc4f40a2034d48a6a236e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:28:41.150358 30286 raft_consensus.cc:2258] T 3cdf482209a8484783087c17fc402263 P 22c4bae1f7fc4f40a2034d48a6a236e4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260923 17:28:41.151038 29780 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.229.1:45107 (error 108)
I20260923 17:28:41.152150 30256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:41.151835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:45107 (local address 127.0.0.1:49470)
0923 17:28:41.151969 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:41.152045 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.1:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260923 17:28:41.152230 30286 ts_tablet_manager.cc:1905] T 3cdf482209a8484783087c17fc402263 P 22c4bae1f7fc4f40a2034d48a6a236e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:41.152288 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 29985
W20260923 17:28:41.153827 29780 catalog_manager.cc:4584] TS 1945e6406c1146cfa755bcfef3d842b8 (127.11.229.1:45107): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3cdf482209a8484783087c17fc402263: Network error: Client connection negotiation failed: client connection to 127.11.229.1:45107: connect: Connection refused (error 111)
I20260923 17:28:41.156772 30286 ts_tablet_manager.cc:1918] T 3cdf482209a8484783087c17fc402263 P 22c4bae1f7fc4f40a2034d48a6a236e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:28:41.156985 30286 log.cc:1196] T 3cdf482209a8484783087c17fc402263 P 22c4bae1f7fc4f40a2034d48a6a236e4: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184271597484-12180-0/minicluster-data/ts-2/wal/wals/3cdf482209a8484783087c17fc402263
I20260923 17:28:41.157541 30286 ts_tablet_manager.cc:1939] T 3cdf482209a8484783087c17fc402263 P 22c4bae1f7fc4f40a2034d48a6a236e4: Deleting consensus metadata
I20260923 17:28:41.159401 29780 catalog_manager.cc:4841] TS 22c4bae1f7fc4f40a2034d48a6a236e4 (127.11.229.3:35131): tablet 3cdf482209a8484783087c17fc402263 (table test-table [id=3df3fd9fc1894c27bdddbbe30c53b72f]) successfully deleted
I20260923 17:28:41.159750 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 30116
I20260923 17:28:41.166131 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 29763
2026-09-23T17:28:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1258 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:28:41 dist-test-slave-ptl9 krb5kdc[30290](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:28:41 dist-test-slave-ptl9 krb5kdc[30290](info): set up 2 sockets
Sep 23 17:28:41 dist-test-slave-ptl9 krb5kdc[30290](info): commencing operation
krb5kdc: starting...
W20260923 17:28:42.225559 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:28:42 dist-test-slave-ptl9 krb5kdc[30290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:28:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:28:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.229.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
I20260923 17:28:42.356274 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:45211
--webserver_interface=127.11.229.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:36715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:45211
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:42.480300 30303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:42.480665 30303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:42.480741 30303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:42.488452 30303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:28:42.488543 30303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:42.488600 30303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:42.488632 30303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:28:42.488662 30303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:28:42.495149 30303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:36715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:45211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:45211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:42.496389 30303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:42.497598 30303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 17:28:42.506759 30303 server_base.cc:1034] running on GCE node
W20260923 17:28:42.506671 30308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:42.506712 30311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:42.506671 30309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:42.507526 30303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:42.508693 30303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:42.509938 30303 hybrid_clock.cc:648] HybridClock initialized: now 1790184522509905 us; error 42 us; skew 500 ppm
Sep 23 17:28:42 dist-test-slave-ptl9 krb5kdc[30290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184522, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:42.512884 30303 init.cc:377] Logged in from keytab as kudu/127.11.229.62@KRBTEST.COM (short username kudu)
I20260923 17:28:42.513898 30303 webserver.cc:460] Webserver started at http://127.11.229.62:44097/ using document root <none> and password file <none>
I20260923 17:28:42.514463 30303 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:42.514523 30303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:42.514694 30303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:42.516795 30303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "3a5d4512441f4aa6a19a564d56352e88"
format_stamp: "Formatted at 2026-09-23 17:28:42 on dist-test-slave-ptl9"
server_key: "be167e2ab108c0eb5c9077ae93c1b3ac"
server_key_iv: "6ff98a137214261d65aee48a07237c75"
server_key_version: "encryptionkey@0"
I20260923 17:28:42.517391 30303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "3a5d4512441f4aa6a19a564d56352e88"
format_stamp: "Formatted at 2026-09-23 17:28:42 on dist-test-slave-ptl9"
server_key: "be167e2ab108c0eb5c9077ae93c1b3ac"
server_key_iv: "6ff98a137214261d65aee48a07237c75"
server_key_version: "encryptionkey@0"
I20260923 17:28:42.521353 30303 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:28:42.523947 30318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:42.525377 30303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:28:42.525490 30303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "3a5d4512441f4aa6a19a564d56352e88"
format_stamp: "Formatted at 2026-09-23 17:28:42 on dist-test-slave-ptl9"
server_key: "be167e2ab108c0eb5c9077ae93c1b3ac"
server_key_iv: "6ff98a137214261d65aee48a07237c75"
server_key_version: "encryptionkey@0"
I20260923 17:28:42.525604 30303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:42.556555 30303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:42.560070 30303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:42.560240 30303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:42.567613 30303 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:45211
I20260923 17:28:42.567631 30370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:45211 every 8 connection(s)
I20260923 17:28:42.568598 30303 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:28:42.571492 30371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:42.573621 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 30303
I20260923 17:28:42.573742 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:28:42.574018 12180 external_mini_cluster.cc:1428] Setting key 943c54009b22eac176ba5d84b9eb9986
I20260923 17:28:42.577590 30371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a5d4512441f4aa6a19a564d56352e88: Bootstrap starting.
I20260923 17:28:42.579854 30371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a5d4512441f4aa6a19a564d56352e88: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:42.580603 30371 log.cc:824] T 00000000000000000000000000000000 P 3a5d4512441f4aa6a19a564d56352e88: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:42.582540 30371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a5d4512441f4aa6a19a564d56352e88: No bootstrap required, opened a new log
Sep 23 17:28:42 dist-test-slave-ptl9 krb5kdc[30290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:28:42.586709 30371 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a5d4512441f4aa6a19a564d56352e88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5d4512441f4aa6a19a564d56352e88" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 45211 } }
I20260923 17:28:42.587041 30371 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a5d4512441f4aa6a19a564d56352e88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:42.587098 30371 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a5d4512441f4aa6a19a564d56352e88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a5d4512441f4aa6a19a564d56352e88, State: Initialized, Role: FOLLOWER
I20260923 17:28:42.587600 30371 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a5d4512441f4aa6a19a564d56352e88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5d4512441f4aa6a19a564d56352e88" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 45211 } }
I20260923 17:28:42.587762 30371 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a5d4512441f4aa6a19a564d56352e88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:28:42.587847 30371 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a5d4512441f4aa6a19a564d56352e88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:28:42.587967 30371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a5d4512441f4aa6a19a564d56352e88 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:42.588196 30374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:42.575416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57810 (local address 127.11.229.62:45211)
0923 17:28:42.575855 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:42.575872 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:42.575906 (+ 34us) server_negotiation.cc:407] Connection header received
0923 17:28:42.576642 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:42.576661 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:42.577002 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:42.577355 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:42.578182 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:42.579170 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:42.580073 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:42.580533 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:42.580704 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:42.583247 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:42.583252 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:42.583266 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:42.583293 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:42.585010 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:42.585576 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:42.585582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:42.585587 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:42.585719 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:42.586013 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:42.586015 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:42.586016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:42.586423 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:42.586594 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:42.586974 (+ 380us) server_negotiation.cc:299] Negotiation successful
0923 17:28:42.587306 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":138,"threads_started":1}
I20260923 17:28:42.589015 30371 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a5d4512441f4aa6a19a564d56352e88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5d4512441f4aa6a19a564d56352e88" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 45211 } }
I20260923 17:28:42.589437 30371 leader_election.cc:302] T 00000000000000000000000000000000 P 3a5d4512441f4aa6a19a564d56352e88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a5d4512441f4aa6a19a564d56352e88; no voters:
I20260923 17:28:42.589813 30371 leader_election.cc:288] T 00000000000000000000000000000000 P 3a5d4512441f4aa6a19a564d56352e88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:28:42.589918 30376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a5d4512441f4aa6a19a564d56352e88 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:42.590129 30376 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a5d4512441f4aa6a19a564d56352e88 [term 1 LEADER]: Becoming Leader. State: Replica: 3a5d4512441f4aa6a19a564d56352e88, State: Running, Role: LEADER
I20260923 17:28:42.590545 30376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a5d4512441f4aa6a19a564d56352e88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5d4512441f4aa6a19a564d56352e88" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 45211 } }
I20260923 17:28:42.591076 30371 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a5d4512441f4aa6a19a564d56352e88 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:28:42.591710 30376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a5d4512441f4aa6a19a564d56352e88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a5d4512441f4aa6a19a564d56352e88. Latest consensus state: current_term: 1 leader_uuid: "3a5d4512441f4aa6a19a564d56352e88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5d4512441f4aa6a19a564d56352e88" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 45211 } } }
I20260923 17:28:42.591826 30376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a5d4512441f4aa6a19a564d56352e88 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:42.592502 30382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:28:42.592854 30377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a5d4512441f4aa6a19a564d56352e88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a5d4512441f4aa6a19a564d56352e88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5d4512441f4aa6a19a564d56352e88" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 45211 } } }
I20260923 17:28:42.593101 30377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a5d4512441f4aa6a19a564d56352e88 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:42.596086 30382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:28:42.602619 30382 catalog_manager.cc:1347] Generated new cluster ID: f484028723d34007b9f297f7bf75e8ef
I20260923 17:28:42.602691 30382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:28:42.608099 30382 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:28:42.609356 30382 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:28:42.622428 30382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a5d4512441f4aa6a19a564d56352e88: Generated new TSK 0
I20260923 17:28:42.623513 30382 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.229.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab.
Entry for principal kudu/127.11.229.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab.
I20260923 17:28:42.664915 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.1:0
--local_ip_for_outbound_sockets=127.11.229.1
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:45211
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:36715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:42.785550 30397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:42.785832 30397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:42.785876 30397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:42.789952 30397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:42.790014 30397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:42.790072 30397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.1
I20260923 17:28:42.796277 30397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:36715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:45211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.1
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:42.797395 30397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:42.798393 30397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:42.808521 30402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:42.808691 30397 server_base.cc:1034] running on GCE node
W20260923 17:28:42.808525 30405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:42.808521 30403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:42.809252 30397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:42.809976 30397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:42.811167 30397 hybrid_clock.cc:648] HybridClock initialized: now 1790184522811146 us; error 29 us; skew 500 ppm
Sep 23 17:28:42 dist-test-slave-ptl9 krb5kdc[30290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184522, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:42.814162 30397 init.cc:377] Logged in from keytab as kudu/127.11.229.1@KRBTEST.COM (short username kudu)
I20260923 17:28:42.815117 30397 webserver.cc:460] Webserver started at http://127.11.229.1:41961/ using document root <none> and password file <none>
I20260923 17:28:42.815693 30397 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:42.815757 30397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:42.815932 30397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:42.818145 30397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-0/data/instance:
uuid: "0c03d7c898a8444ab6fd9e89ea1f033d"
format_stamp: "Formatted at 2026-09-23 17:28:42 on dist-test-slave-ptl9"
server_key: "5564aed81872abde9f116923dc1d9eda"
server_key_iv: "569ade632cf87891efdd54f529633372"
server_key_version: "encryptionkey@0"
I20260923 17:28:42.818691 30397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance:
uuid: "0c03d7c898a8444ab6fd9e89ea1f033d"
format_stamp: "Formatted at 2026-09-23 17:28:42 on dist-test-slave-ptl9"
server_key: "5564aed81872abde9f116923dc1d9eda"
server_key_iv: "569ade632cf87891efdd54f529633372"
server_key_version: "encryptionkey@0"
I20260923 17:28:42.822774 30397 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:28:42.825467 30412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:42.826874 30397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:28:42.827024 30397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-0/wal
uuid: "0c03d7c898a8444ab6fd9e89ea1f033d"
format_stamp: "Formatted at 2026-09-23 17:28:42 on dist-test-slave-ptl9"
server_key: "5564aed81872abde9f116923dc1d9eda"
server_key_iv: "569ade632cf87891efdd54f529633372"
server_key_version: "encryptionkey@0"
I20260923 17:28:42.827150 30397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:42.840781 30397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:42.844357 30397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:42.844578 30397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:42.845216 30397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:42.846156 30397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:42.846226 30397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:42.846298 30397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:42.846338 30397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:42.856310 30397 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.1:34715
I20260923 17:28:42.856335 30525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.1:34715 every 8 connection(s)
I20260923 17:28:42.857309 30397 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
I20260923 17:28:42.862489 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 30397
I20260923 17:28:42.862627 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance
I20260923 17:28:42.862953 12180 external_mini_cluster.cc:1428] Setting key 7f4e84f2325881f4b53b4309f637b4f0
Sep 23 17:28:42 dist-test-slave-ptl9 krb5kdc[30290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184522, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.229.2@KRBTEST.COM; defaulting to no policy
I20260923 17:28:42.872421 30374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:42.859303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:57235 (local address 127.11.229.62:45211)
0923 17:28:42.859495 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:42.859500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:42.860333 (+ 833us) server_negotiation.cc:407] Connection header received
0923 17:28:42.861680 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:42.861683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:42.861733 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:42.861808 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:42.863426 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:42.864109 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:42.865372 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:42.865785 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:42.865913 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:42.868792 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:42.868795 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:42.868797 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:42.868822 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:42.870271 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:42.870814 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:42.870818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:42.870820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:42.870906 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:42.871251 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:42.871256 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:42.871258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:42.871460 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:42.871565 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:42.872100 (+ 535us) server_negotiation.cc:299] Negotiation successful
0923 17:28:42.872235 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260923 17:28:42.873375 30528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:42.859703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:45211 (local address 127.11.229.1:57235)
0923 17:28:42.860174 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:42.860219 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:42.861480 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:42.861981 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:42.861987 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:42.862491 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:42.863211 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:42.863233 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:42.864248 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:42.864252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:42.865177 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:42.865187 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:42.865981 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:42.865988 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:42.866232 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:42.866923 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:42.866948 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:42.868604 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:42.870409 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:42.870415 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:42.870433 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:42.870706 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:42.870999 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:42.871002 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:42.871004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:42.871154 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:42.871567 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:42.871575 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:42.871844 (+ 269us) client_negotiation.cc:769] Sending connection context
0923 17:28:42.872055 (+ 211us) client_negotiation.cc:240] Negotiation successful
0923 17:28:42.872408 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":136,"threads_started":1}
I20260923 17:28:42.874751 30526 heartbeater.cc:347] Connected to a master server at 127.11.229.62:45211
I20260923 17:28:42.875041 30526 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.2@KRBTEST.COM" created.
I20260923 17:28:42.875661 30526 heartbeater.cc:511] Master 127.11.229.62:45211 requested a full tablet report, sending...
I20260923 17:28:42.877514 30335 ts_manager.cc:194] Registered new tserver with Master: 0c03d7c898a8444ab6fd9e89ea1f033d (127.11.229.1:34715)
I20260923 17:28:42.878669 30335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.1@KRBTEST.COM'} at 127.11.229.1:57235
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab.
Entry for principal kudu/127.11.229.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab.
I20260923 17:28:42.893440 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.2:0
--local_ip_for_outbound_sockets=127.11.229.2
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:45211
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:36715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:43.009892 30531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:43.010223 30531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:43.010301 30531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:43.014542 30531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:43.014645 30531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:43.014741 30531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.2
I20260923 17:28:43.020329 30531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:36715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:45211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.2
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:43.021641 30531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:43.022737 30531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:43.033247 30537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:43.033210 30536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:43.033210 30539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:43.033437 30531 server_base.cc:1034] running on GCE node
I20260923 17:28:43.034016 30531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:43.034655 30531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:43.035841 30531 hybrid_clock.cc:648] HybridClock initialized: now 1790184523035818 us; error 44 us; skew 500 ppm
Sep 23 17:28:43 dist-test-slave-ptl9 krb5kdc[30290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184523, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:43.038707 30531 init.cc:377] Logged in from keytab as kudu/127.11.229.2@KRBTEST.COM (short username kudu)
I20260923 17:28:43.039626 30531 webserver.cc:460] Webserver started at http://127.11.229.2:43365/ using document root <none> and password file <none>
I20260923 17:28:43.040199 30531 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:43.040266 30531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:43.040438 30531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:43.042568 30531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-1/data/instance:
uuid: "cf38d61408bd4eada8f8ac705a91fc20"
format_stamp: "Formatted at 2026-09-23 17:28:43 on dist-test-slave-ptl9"
server_key: "83fe63b0cb354fe24ffc6b0eb1127016"
server_key_iv: "f41a68e75dbf4ef942c9a837337ba88c"
server_key_version: "encryptionkey@0"
I20260923 17:28:43.043084 30531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance:
uuid: "cf38d61408bd4eada8f8ac705a91fc20"
format_stamp: "Formatted at 2026-09-23 17:28:43 on dist-test-slave-ptl9"
server_key: "83fe63b0cb354fe24ffc6b0eb1127016"
server_key_iv: "f41a68e75dbf4ef942c9a837337ba88c"
server_key_version: "encryptionkey@0"
I20260923 17:28:43.047381 30531 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:28:43.050024 30546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:43.051388 30531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:28:43.051546 30531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-1/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-1/wal
uuid: "cf38d61408bd4eada8f8ac705a91fc20"
format_stamp: "Formatted at 2026-09-23 17:28:43 on dist-test-slave-ptl9"
server_key: "83fe63b0cb354fe24ffc6b0eb1127016"
server_key_iv: "f41a68e75dbf4ef942c9a837337ba88c"
server_key_version: "encryptionkey@0"
I20260923 17:28:43.051676 30531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:43.063791 30531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:43.067190 30531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:43.067407 30531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:43.068027 30531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:43.068922 30531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:43.068969 30531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:43.069029 30531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:43.069061 30531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:43.078145 30531 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.2:35415
I20260923 17:28:43.078162 30659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.2:35415 every 8 connection(s)
I20260923 17:28:43.079077 30531 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
I20260923 17:28:43.080228 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 30531
I20260923 17:28:43.080386 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance
I20260923 17:28:43.080770 12180 external_mini_cluster.cc:1428] Setting key a9d4499ae11f65c865d641249b385a3c
Sep 23 17:28:43 dist-test-slave-ptl9 krb5kdc[30290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184523, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.229.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.3@KRBTEST.COM" created.
I20260923 17:28:43.095712 30374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:43.081226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:51061 (local address 127.11.229.62:45211)
0923 17:28:43.081438 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:43.081444 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:43.082438 (+ 994us) server_negotiation.cc:407] Connection header received
0923 17:28:43.083887 (+ 1449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:43.083893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:43.083958 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:43.084041 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:43.085844 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.086379 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.088437 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.088658 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.088755 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:43.091912 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:43.091916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:43.091918 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:43.091942 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:43.093568 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:43.094087 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:43.094095 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:43.094097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:43.094170 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:43.094521 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:43.094525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:43.094526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:43.094716 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:43.094827 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:43.095450 (+ 623us) server_negotiation.cc:299] Negotiation successful
0923 17:28:43.095571 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 17:28:43.096925 30662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:43.081720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:45211 (local address 127.11.229.2:51061)
0923 17:28:43.082283 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:43.082347 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:43.083630 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:43.084280 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:43.084292 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:43.084850 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:43.085680 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:43.085705 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.086520 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.086527 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:43.088298 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:43.088314 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.088833 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.088841 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:43.089099 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:43.089827 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:43.089854 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:43.091689 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:43.093688 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:43.093695 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:43.093712 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:43.093974 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:43.094268 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:43.094271 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:43.094273 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:43.094421 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:43.094821 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:43.094829 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:43.095127 (+ 298us) client_negotiation.cc:769] Sending connection context
0923 17:28:43.095366 (+ 239us) client_negotiation.cc:240] Negotiation successful
0923 17:28:43.095758 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":127,"threads_started":1}
I20260923 17:28:43.098326 30660 heartbeater.cc:347] Connected to a master server at 127.11.229.62:45211
I20260923 17:28:43.098611 30660 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:43.099298 30660 heartbeater.cc:511] Master 127.11.229.62:45211 requested a full tablet report, sending...
I20260923 17:28:43.100555 30335 ts_manager.cc:194] Registered new tserver with Master: cf38d61408bd4eada8f8ac705a91fc20 (127.11.229.2:35415)
I20260923 17:28:43.101143 30335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.2@KRBTEST.COM'} at 127.11.229.2:51061
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab.
Entry for principal kudu/127.11.229.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab.
I20260923 17:28:43.112636 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.3:0
--local_ip_for_outbound_sockets=127.11.229.3
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:45211
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:36715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:43.225173 30665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:43.225489 30665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:43.225582 30665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:43.230012 30665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:43.230098 30665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:43.230195 30665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.3
I20260923 17:28:43.237102 30665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:36715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:45211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.3
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:43.238266 30665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:43.239333 30665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:43.249290 30673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:43.249321 30671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:43.249310 30670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:43.249620 30665 server_base.cc:1034] running on GCE node
I20260923 17:28:43.250046 30665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:43.250715 30665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:43.251900 30665 hybrid_clock.cc:648] HybridClock initialized: now 1790184523251885 us; error 29 us; skew 500 ppm
Sep 23 17:28:43 dist-test-slave-ptl9 krb5kdc[30290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184523, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:43.255096 30665 init.cc:377] Logged in from keytab as kudu/127.11.229.3@KRBTEST.COM (short username kudu)
I20260923 17:28:43.256042 30665 webserver.cc:460] Webserver started at http://127.11.229.3:34849/ using document root <none> and password file <none>
I20260923 17:28:43.256619 30665 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:43.256708 30665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:43.256924 30665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:43.259088 30665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-2/data/instance:
uuid: "eb8f9ba051bd43b1abce006d158819ea"
format_stamp: "Formatted at 2026-09-23 17:28:43 on dist-test-slave-ptl9"
server_key: "e164f0014571d462b13f8eab009a2d06"
server_key_iv: "9aa1e60a4d20f54cdc0357b0904ddd9b"
server_key_version: "encryptionkey@0"
I20260923 17:28:43.259676 30665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance:
uuid: "eb8f9ba051bd43b1abce006d158819ea"
format_stamp: "Formatted at 2026-09-23 17:28:43 on dist-test-slave-ptl9"
server_key: "e164f0014571d462b13f8eab009a2d06"
server_key_iv: "9aa1e60a4d20f54cdc0357b0904ddd9b"
server_key_version: "encryptionkey@0"
I20260923 17:28:43.263463 30665 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:28:43.266050 30680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:43.267355 30665 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:28:43.267489 30665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-2/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-2/wal
uuid: "eb8f9ba051bd43b1abce006d158819ea"
format_stamp: "Formatted at 2026-09-23 17:28:43 on dist-test-slave-ptl9"
server_key: "e164f0014571d462b13f8eab009a2d06"
server_key_iv: "9aa1e60a4d20f54cdc0357b0904ddd9b"
server_key_version: "encryptionkey@0"
I20260923 17:28:43.267602 30665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:43.289028 30665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:43.292295 30665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:43.292496 30665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:43.293159 30665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:43.294044 30665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:43.294119 30665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:43.294193 30665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:43.294241 30665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:43.304286 30665 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.3:35461
I20260923 17:28:43.304318 30793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.3:35461 every 8 connection(s)
I20260923 17:28:43.305832 30665 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
I20260923 17:28:43.309751 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 30665
I20260923 17:28:43.309893 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance
I20260923 17:28:43.310230 12180 external_mini_cluster.cc:1428] Setting key cb4eda2b6f5bfe489b15a4812ab0072c
Sep 23 17:28:43 dist-test-slave-ptl9 krb5kdc[30290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184523, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:28:43.321888 30374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:43.308254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:34715 (local address 127.11.229.62:45211)
0923 17:28:43.308397 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:43.308402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:43.309717 (+ 1315us) server_negotiation.cc:407] Connection header received
0923 17:28:43.311639 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:43.311643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:43.311707 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:43.311783 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:43.313321 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.313884 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.315099 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.315352 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.315548 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:43.318321 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:43.318325 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:43.318327 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:43.318357 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:43.319663 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:43.320193 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:43.320197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:43.320199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:43.320282 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:43.320636 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:43.320639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:43.320641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:43.320848 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:43.320950 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:43.321561 (+ 611us) server_negotiation.cc:299] Negotiation successful
0923 17:28:43.321699 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 17:28:43.322759 30796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:43.308852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:45211 (local address 127.11.229.3:34715)
0923 17:28:43.309532 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:43.309595 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:43.311337 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:43.311966 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:43.311975 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:43.312442 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:43.313134 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:43.313156 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.314014 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.314018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:43.314970 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:43.314980 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.315587 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.315591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:43.315788 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:43.316529 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:43.316546 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:43.318136 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:43.319790 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:43.319796 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:43.319815 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:43.320081 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:43.320377 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:43.320382 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:43.320384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:43.320536 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:43.320965 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:43.320973 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:43.321240 (+ 267us) client_negotiation.cc:769] Sending connection context
0923 17:28:43.321506 (+ 266us) client_negotiation.cc:240] Negotiation successful
0923 17:28:43.321871 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":174,"threads_started":1}
I20260923 17:28:43.323983 30794 heartbeater.cc:347] Connected to a master server at 127.11.229.62:45211
I20260923 17:28:43.324226 30794 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:43.324748 30794 heartbeater.cc:511] Master 127.11.229.62:45211 requested a full tablet report, sending...
I20260923 17:28:43.326095 30335 ts_manager.cc:194] Registered new tserver with Master: eb8f9ba051bd43b1abce006d158819ea (127.11.229.3:35461)
I20260923 17:28:43.326687 30335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.3@KRBTEST.COM'} at 127.11.229.3:34715
I20260923 17:28:43.337195 12180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:28:43.347169 30374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:43.339218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57820 (local address 127.11.229.62:45211)
0923 17:28:43.339371 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:43.339376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:43.339556 (+ 180us) server_negotiation.cc:407] Connection header received
0923 17:28:43.339720 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:43.339724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:43.339766 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:43.339856 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:43.340576 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.341126 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.342006 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.342227 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.342334 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:43.343968 (+ 1634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:43.343971 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:43.343974 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:43.343997 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:43.345491 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:43.346009 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:43.346012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:43.346014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:43.346120 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:43.346429 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:43.346432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:43.346433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:43.346643 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:43.346756 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:43.346959 (+ 203us) server_negotiation.cc:299] Negotiation successful
0923 17:28:43.347043 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 17:28:43.350238 30335 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:28:43.353433 30335 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:28:43.368952 30806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:43.362411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:35415 (local address 127.0.0.1:60450)
0923 17:28:43.362992 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:43.363047 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:43.363184 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:43.364017 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:43.364023 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:43.364049 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:43.364280 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:43.364287 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.365903 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.365907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:43.367240 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:43.367251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.368289 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.368296 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:43.368410 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:43.368450 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 17:28:43.368563 (+ 113us) client_negotiation.cc:240] Negotiation successful
0923 17:28:43.368689 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":34,"thread_start_us":84,"threads_started":1}
I20260923 17:28:43.369078 30809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:43.363684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:35461 (local address 127.0.0.1:53764)
0923 17:28:43.364477 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:43.364488 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:43.364599 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:43.365025 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:43.365028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:43.365045 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:43.365136 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:43.365143 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.366455 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.366458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:43.367527 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:43.367535 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.368540 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.368546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:43.368641 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:43.368660 (+ 19us) client_negotiation.cc:769] Sending connection context
0923 17:28:43.368731 (+ 71us) client_negotiation.cc:240] Negotiation successful
0923 17:28:43.368907 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"thread_start_us":91,"threads_started":1}
I20260923 17:28:43.369545 30807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:43.362588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60450 (local address 127.11.229.2:35415)
0923 17:28:43.363494 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:43.363507 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:43.363532 (+ 25us) server_negotiation.cc:407] Connection header received
0923 17:28:43.363618 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:43.363623 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:43.363835 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:43.364441 (+ 606us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:43.364602 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.365741 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.367365 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.368146 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.368263 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:43.368417 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:43.368611 (+ 194us) server_negotiation.cc:299] Negotiation successful
0923 17:28:43.368694 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":777,"thread_start_us":132,"threads_started":1}
I20260923 17:28:43.369897 30805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:43.362597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:34715 (local address 127.0.0.1:36720)
0923 17:28:43.363256 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:43.363271 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:43.363369 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:43.364172 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:43.364177 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:43.364191 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:43.364280 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:43.364287 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.365809 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.365813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:43.367143 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:43.367160 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.369404 (+ 2244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.369410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:43.369552 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:43.369594 (+ 42us) client_negotiation.cc:769] Sending connection context
0923 17:28:43.369679 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 17:28:43.369758 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"mutex_wait_us":50,"thread_start_us":119,"threads_started":1}
I20260923 17:28:43.370333 30808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:43.363213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36720 (local address 127.11.229.1:34715)
0923 17:28:43.363554 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:43.363568 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:43.363586 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:28:43.363651 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:43.363655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:43.363835 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:43.363997 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:43.364418 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.365666 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.367267 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.368146 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.368263 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:43.368417 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:43.369799 (+ 1382us) server_negotiation.cc:299] Negotiation successful
0923 17:28:43.370073 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":146,"threads_started":1}
I20260923 17:28:43.371181 30810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:43.364073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53764 (local address 127.11.229.3:35461)
0923 17:28:43.364354 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:43.364365 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:43.364577 (+ 212us) server_negotiation.cc:407] Connection header received
0923 17:28:43.364727 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:43.364732 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:43.364878 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:43.365048 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:43.365230 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.366160 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.367639 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.368197 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.368329 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:43.368477 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:43.368943 (+ 466us) server_negotiation.cc:299] Negotiation successful
0923 17:28:43.369055 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":99,"threads_started":1}
I20260923 17:28:43.373004 30460 tablet_service.cc:1467] Processing CreateTablet for tablet d8f18b93328f44a39e9431ecf696146e (DEFAULT_TABLE table=test-table [id=0ffd02549b63406188e41de7cc120137]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:43.373004 30594 tablet_service.cc:1467] Processing CreateTablet for tablet d8f18b93328f44a39e9431ecf696146e (DEFAULT_TABLE table=test-table [id=0ffd02549b63406188e41de7cc120137]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:43.373512 30728 tablet_service.cc:1467] Processing CreateTablet for tablet d8f18b93328f44a39e9431ecf696146e (DEFAULT_TABLE table=test-table [id=0ffd02549b63406188e41de7cc120137]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:43.374236 30460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8f18b93328f44a39e9431ecf696146e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:43.374236 30594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8f18b93328f44a39e9431ecf696146e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:43.374560 30728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8f18b93328f44a39e9431ecf696146e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:43.382408 30811 tablet_bootstrap.cc:492] T d8f18b93328f44a39e9431ecf696146e P cf38d61408bd4eada8f8ac705a91fc20: Bootstrap starting.
I20260923 17:28:43.385449 30811 tablet_bootstrap.cc:654] T d8f18b93328f44a39e9431ecf696146e P cf38d61408bd4eada8f8ac705a91fc20: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:43.385660 30812 tablet_bootstrap.cc:492] T d8f18b93328f44a39e9431ecf696146e P 0c03d7c898a8444ab6fd9e89ea1f033d: Bootstrap starting.
I20260923 17:28:43.386478 30811 log.cc:824] T d8f18b93328f44a39e9431ecf696146e P cf38d61408bd4eada8f8ac705a91fc20: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:43.388283 30812 tablet_bootstrap.cc:654] T d8f18b93328f44a39e9431ecf696146e P 0c03d7c898a8444ab6fd9e89ea1f033d: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:43.388926 30811 tablet_bootstrap.cc:492] T d8f18b93328f44a39e9431ecf696146e P cf38d61408bd4eada8f8ac705a91fc20: No bootstrap required, opened a new log
I20260923 17:28:43.389142 30811 ts_tablet_manager.cc:1397] T d8f18b93328f44a39e9431ecf696146e P cf38d61408bd4eada8f8ac705a91fc20: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260923 17:28:43.389246 30812 log.cc:824] T d8f18b93328f44a39e9431ecf696146e P 0c03d7c898a8444ab6fd9e89ea1f033d: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:43.390102 30813 tablet_bootstrap.cc:492] T d8f18b93328f44a39e9431ecf696146e P eb8f9ba051bd43b1abce006d158819ea: Bootstrap starting.
I20260923 17:28:43.391577 30812 tablet_bootstrap.cc:492] T d8f18b93328f44a39e9431ecf696146e P 0c03d7c898a8444ab6fd9e89ea1f033d: No bootstrap required, opened a new log
I20260923 17:28:43.391839 30812 ts_tablet_manager.cc:1397] T d8f18b93328f44a39e9431ecf696146e P 0c03d7c898a8444ab6fd9e89ea1f033d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.006s
I20260923 17:28:43.393400 30813 tablet_bootstrap.cc:654] T d8f18b93328f44a39e9431ecf696146e P eb8f9ba051bd43b1abce006d158819ea: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:43.394460 30813 log.cc:824] T d8f18b93328f44a39e9431ecf696146e P eb8f9ba051bd43b1abce006d158819ea: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:43.395750 30811 raft_consensus.cc:348] T d8f18b93328f44a39e9431ecf696146e P cf38d61408bd4eada8f8ac705a91fc20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf38d61408bd4eada8f8ac705a91fc20" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 35415 } } peers { permanent_uuid: "0c03d7c898a8444ab6fd9e89ea1f033d" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34715 } } peers { permanent_uuid: "eb8f9ba051bd43b1abce006d158819ea" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35461 } }
I20260923 17:28:43.396144 30811 raft_consensus.cc:374] T d8f18b93328f44a39e9431ecf696146e P cf38d61408bd4eada8f8ac705a91fc20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:43.396224 30811 raft_consensus.cc:729] T d8f18b93328f44a39e9431ecf696146e P cf38d61408bd4eada8f8ac705a91fc20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf38d61408bd4eada8f8ac705a91fc20, State: Initialized, Role: FOLLOWER
I20260923 17:28:43.396723 30813 tablet_bootstrap.cc:492] T d8f18b93328f44a39e9431ecf696146e P eb8f9ba051bd43b1abce006d158819ea: No bootstrap required, opened a new log
I20260923 17:28:43.396698 30811 consensus_queue.cc:260] T d8f18b93328f44a39e9431ecf696146e P cf38d61408bd4eada8f8ac705a91fc20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf38d61408bd4eada8f8ac705a91fc20" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 35415 } } peers { permanent_uuid: "0c03d7c898a8444ab6fd9e89ea1f033d" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34715 } } peers { permanent_uuid: "eb8f9ba051bd43b1abce006d158819ea" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35461 } }
I20260923 17:28:43.396972 30813 ts_tablet_manager.cc:1397] T d8f18b93328f44a39e9431ecf696146e P eb8f9ba051bd43b1abce006d158819ea: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260923 17:28:43.397032 30812 raft_consensus.cc:348] T d8f18b93328f44a39e9431ecf696146e P 0c03d7c898a8444ab6fd9e89ea1f033d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf38d61408bd4eada8f8ac705a91fc20" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 35415 } } peers { permanent_uuid: "0c03d7c898a8444ab6fd9e89ea1f033d" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34715 } } peers { permanent_uuid: "eb8f9ba051bd43b1abce006d158819ea" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35461 } }
I20260923 17:28:43.397403 30812 raft_consensus.cc:374] T d8f18b93328f44a39e9431ecf696146e P 0c03d7c898a8444ab6fd9e89ea1f033d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:43.397481 30812 raft_consensus.cc:729] T d8f18b93328f44a39e9431ecf696146e P 0c03d7c898a8444ab6fd9e89ea1f033d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c03d7c898a8444ab6fd9e89ea1f033d, State: Initialized, Role: FOLLOWER
I20260923 17:28:43.397593 30660 heartbeater.cc:503] Master 127.11.229.62:45211 was elected leader, sending a full tablet report...
I20260923 17:28:43.397938 30812 consensus_queue.cc:260] T d8f18b93328f44a39e9431ecf696146e P 0c03d7c898a8444ab6fd9e89ea1f033d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf38d61408bd4eada8f8ac705a91fc20" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 35415 } } peers { permanent_uuid: "0c03d7c898a8444ab6fd9e89ea1f033d" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34715 } } peers { permanent_uuid: "eb8f9ba051bd43b1abce006d158819ea" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35461 } }
I20260923 17:28:43.398016 30811 ts_tablet_manager.cc:1428] T d8f18b93328f44a39e9431ecf696146e P cf38d61408bd4eada8f8ac705a91fc20: Time spent starting tablet: real 0.009s user 0.000s sys 0.007s
I20260923 17:28:43.398939 30812 ts_tablet_manager.cc:1428] T d8f18b93328f44a39e9431ecf696146e P 0c03d7c898a8444ab6fd9e89ea1f033d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260923 17:28:43.399307 30526 heartbeater.cc:503] Master 127.11.229.62:45211 was elected leader, sending a full tablet report...
I20260923 17:28:43.402498 30813 raft_consensus.cc:348] T d8f18b93328f44a39e9431ecf696146e P eb8f9ba051bd43b1abce006d158819ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf38d61408bd4eada8f8ac705a91fc20" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 35415 } } peers { permanent_uuid: "0c03d7c898a8444ab6fd9e89ea1f033d" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34715 } } peers { permanent_uuid: "eb8f9ba051bd43b1abce006d158819ea" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35461 } }
I20260923 17:28:43.402858 30813 raft_consensus.cc:374] T d8f18b93328f44a39e9431ecf696146e P eb8f9ba051bd43b1abce006d158819ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:43.402927 30813 raft_consensus.cc:729] T d8f18b93328f44a39e9431ecf696146e P eb8f9ba051bd43b1abce006d158819ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb8f9ba051bd43b1abce006d158819ea, State: Initialized, Role: FOLLOWER
I20260923 17:28:43.403415 30813 consensus_queue.cc:260] T d8f18b93328f44a39e9431ecf696146e P eb8f9ba051bd43b1abce006d158819ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf38d61408bd4eada8f8ac705a91fc20" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 35415 } } peers { permanent_uuid: "0c03d7c898a8444ab6fd9e89ea1f033d" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34715 } } peers { permanent_uuid: "eb8f9ba051bd43b1abce006d158819ea" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35461 } }
I20260923 17:28:43.404188 30794 heartbeater.cc:503] Master 127.11.229.62:45211 was elected leader, sending a full tablet report...
I20260923 17:28:43.404418 30813 ts_tablet_manager.cc:1428] T d8f18b93328f44a39e9431ecf696146e P eb8f9ba051bd43b1abce006d158819ea: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
W20260923 17:28:43.559732 30795 tablet.cc:2369] T d8f18b93328f44a39e9431ecf696146e P eb8f9ba051bd43b1abce006d158819ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:28:43.575153 30819 raft_consensus.cc:482] T d8f18b93328f44a39e9431ecf696146e P eb8f9ba051bd43b1abce006d158819ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:28:43.575373 30819 raft_consensus.cc:504] T d8f18b93328f44a39e9431ecf696146e P eb8f9ba051bd43b1abce006d158819ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf38d61408bd4eada8f8ac705a91fc20" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 35415 } } peers { permanent_uuid: "0c03d7c898a8444ab6fd9e89ea1f033d" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34715 } } peers { permanent_uuid: "eb8f9ba051bd43b1abce006d158819ea" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35461 } }
I20260923 17:28:43.576673 30819 leader_election.cc:288] T d8f18b93328f44a39e9431ecf696146e P eb8f9ba051bd43b1abce006d158819ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf38d61408bd4eada8f8ac705a91fc20 (127.11.229.2:35415), 0c03d7c898a8444ab6fd9e89ea1f033d (127.11.229.1:34715)
W20260923 17:28:43.580250 30661 tablet.cc:2369] T d8f18b93328f44a39e9431ecf696146e P cf38d61408bd4eada8f8ac705a91fc20: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:28:43.581559 30796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:43.577020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:34715 (local address 127.11.229.3:59135)
0923 17:28:43.577204 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:43.577221 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:43.577360 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:43.577738 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:43.577741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:43.577758 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:43.577872 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:43.577877 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.578876 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.578882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:43.580051 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:43.580063 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.581043 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.581048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:43.581173 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:43.581194 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 17:28:43.581329 (+ 135us) client_negotiation.cc:240] Negotiation successful
0923 17:28:43.581411 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260923 17:28:43.581559 30808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:43.577168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:59135 (local address 127.11.229.1:34715)
0923 17:28:43.577340 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:43.577345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:43.577367 (+ 22us) server_negotiation.cc:407] Connection header received
0923 17:28:43.577514 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:43.577518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:43.577569 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:43.577632 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:43.577990 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.578755 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.580218 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.580914 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.581049 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:43.581111 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:43.581353 (+ 242us) server_negotiation.cc:299] Negotiation successful
0923 17:28:43.581435 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:28:43.581900 30820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:43.577020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:35415 (local address 127.11.229.3:38113)
0923 17:28:43.577744 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:43.577757 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:43.577891 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:43.578263 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:43.578266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:43.578286 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:43.578417 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:43.578423 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.579412 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.579415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:43.580702 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:43.580723 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.581441 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.581445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:43.581551 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:43.581571 (+ 20us) client_negotiation.cc:769] Sending connection context
0923 17:28:43.581677 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 17:28:43.581768 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":366,"thread_start_us":141,"threads_started":1}
I20260923 17:28:43.582199 30807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:43.577341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:38113 (local address 127.11.229.2:35415)
0923 17:28:43.577481 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:43.577488 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:43.577846 (+ 358us) server_negotiation.cc:407] Connection header received
0923 17:28:43.578090 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:43.578093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:43.578145 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:43.578217 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:43.578537 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.579289 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.580857 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.581305 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.581445 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:43.581495 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:43.582031 (+ 536us) server_negotiation.cc:299] Negotiation successful
0923 17:28:43.582111 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 17:28:43.582325 30480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f18b93328f44a39e9431ecf696146e" candidate_uuid: "eb8f9ba051bd43b1abce006d158819ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c03d7c898a8444ab6fd9e89ea1f033d" is_pre_election: true
I20260923 17:28:43.582696 30480 raft_consensus.cc:2454] T d8f18b93328f44a39e9431ecf696146e P 0c03d7c898a8444ab6fd9e89ea1f033d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb8f9ba051bd43b1abce006d158819ea in term 0.
I20260923 17:28:43.582878 30614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f18b93328f44a39e9431ecf696146e" candidate_uuid: "eb8f9ba051bd43b1abce006d158819ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf38d61408bd4eada8f8ac705a91fc20" is_pre_election: true
I20260923 17:28:43.583171 30614 raft_consensus.cc:2454] T d8f18b93328f44a39e9431ecf696146e P cf38d61408bd4eada8f8ac705a91fc20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb8f9ba051bd43b1abce006d158819ea in term 0.
I20260923 17:28:43.583250 30682 leader_election.cc:302] T d8f18b93328f44a39e9431ecf696146e P eb8f9ba051bd43b1abce006d158819ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c03d7c898a8444ab6fd9e89ea1f033d, eb8f9ba051bd43b1abce006d158819ea; no voters:
I20260923 17:28:43.583534 30819 raft_consensus.cc:2781] T d8f18b93328f44a39e9431ecf696146e P eb8f9ba051bd43b1abce006d158819ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:28:43.583614 30819 raft_consensus.cc:482] T d8f18b93328f44a39e9431ecf696146e P eb8f9ba051bd43b1abce006d158819ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:28:43.583678 30819 raft_consensus.cc:3037] T d8f18b93328f44a39e9431ecf696146e P eb8f9ba051bd43b1abce006d158819ea [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:43.584718 30819 raft_consensus.cc:504] T d8f18b93328f44a39e9431ecf696146e P eb8f9ba051bd43b1abce006d158819ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf38d61408bd4eada8f8ac705a91fc20" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 35415 } } peers { permanent_uuid: "0c03d7c898a8444ab6fd9e89ea1f033d" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34715 } } peers { permanent_uuid: "eb8f9ba051bd43b1abce006d158819ea" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35461 } }
I20260923 17:28:43.585094 30819 leader_election.cc:288] T d8f18b93328f44a39e9431ecf696146e P eb8f9ba051bd43b1abce006d158819ea [CANDIDATE]: Term 1 election: Requested vote from peers cf38d61408bd4eada8f8ac705a91fc20 (127.11.229.2:35415), 0c03d7c898a8444ab6fd9e89ea1f033d (127.11.229.1:34715)
I20260923 17:28:43.585489 30614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f18b93328f44a39e9431ecf696146e" candidate_uuid: "eb8f9ba051bd43b1abce006d158819ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf38d61408bd4eada8f8ac705a91fc20"
I20260923 17:28:43.585506 30480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f18b93328f44a39e9431ecf696146e" candidate_uuid: "eb8f9ba051bd43b1abce006d158819ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c03d7c898a8444ab6fd9e89ea1f033d"
I20260923 17:28:43.585628 30614 raft_consensus.cc:3037] T d8f18b93328f44a39e9431ecf696146e P cf38d61408bd4eada8f8ac705a91fc20 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:43.585628 30480 raft_consensus.cc:3037] T d8f18b93328f44a39e9431ecf696146e P 0c03d7c898a8444ab6fd9e89ea1f033d [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:43.586920 30614 raft_consensus.cc:2454] T d8f18b93328f44a39e9431ecf696146e P cf38d61408bd4eada8f8ac705a91fc20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb8f9ba051bd43b1abce006d158819ea in term 1.
I20260923 17:28:43.586920 30480 raft_consensus.cc:2454] T d8f18b93328f44a39e9431ecf696146e P 0c03d7c898a8444ab6fd9e89ea1f033d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb8f9ba051bd43b1abce006d158819ea in term 1.
I20260923 17:28:43.587227 30682 leader_election.cc:302] T d8f18b93328f44a39e9431ecf696146e P eb8f9ba051bd43b1abce006d158819ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0c03d7c898a8444ab6fd9e89ea1f033d, cf38d61408bd4eada8f8ac705a91fc20, eb8f9ba051bd43b1abce006d158819ea; no voters:
I20260923 17:28:43.587379 30819 raft_consensus.cc:2781] T d8f18b93328f44a39e9431ecf696146e P eb8f9ba051bd43b1abce006d158819ea [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:43.587577 30819 raft_consensus.cc:686] T d8f18b93328f44a39e9431ecf696146e P eb8f9ba051bd43b1abce006d158819ea [term 1 LEADER]: Becoming Leader. State: Replica: eb8f9ba051bd43b1abce006d158819ea, State: Running, Role: LEADER
I20260923 17:28:43.587937 30819 consensus_queue.cc:237] T d8f18b93328f44a39e9431ecf696146e P eb8f9ba051bd43b1abce006d158819ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf38d61408bd4eada8f8ac705a91fc20" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 35415 } } peers { permanent_uuid: "0c03d7c898a8444ab6fd9e89ea1f033d" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34715 } } peers { permanent_uuid: "eb8f9ba051bd43b1abce006d158819ea" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35461 } }
I20260923 17:28:43.591135 30335 catalog_manager.cc:5495] T d8f18b93328f44a39e9431ecf696146e P eb8f9ba051bd43b1abce006d158819ea reported cstate change: term changed from 0 to 1, leader changed from <none> to eb8f9ba051bd43b1abce006d158819ea (127.11.229.3). New cstate: current_term: 1 leader_uuid: "eb8f9ba051bd43b1abce006d158819ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf38d61408bd4eada8f8ac705a91fc20" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 35415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c03d7c898a8444ab6fd9e89ea1f033d" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb8f9ba051bd43b1abce006d158819ea" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 35461 } health_report { overall_health: HEALTHY } } }
I20260923 17:28:43.607820 30810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:43.603979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53772 (local address 127.11.229.3:35461)
0923 17:28:43.604120 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:43.604125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:43.604139 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:28:43.604327 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:43.604331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:43.604427 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:43.604544 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:28:43.604983 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.605591 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.606438 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.606644 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.606720 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:43.607055 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:28:43.607462 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:28:43.607570 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:43.607655 (+ 85us) server_negotiation.cc:299] Negotiation successful
0923 17:28:43.607692 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260923 17:28:43.608696 30527 tablet.cc:2369] T d8f18b93328f44a39e9431ecf696146e P 0c03d7c898a8444ab6fd9e89ea1f033d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:28:43.614010 30480 raft_consensus.cc:1261] T d8f18b93328f44a39e9431ecf696146e P 0c03d7c898a8444ab6fd9e89ea1f033d [term 1 FOLLOWER]: Refusing update from remote peer eb8f9ba051bd43b1abce006d158819ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:28:43.614010 30614 raft_consensus.cc:1261] T d8f18b93328f44a39e9431ecf696146e P cf38d61408bd4eada8f8ac705a91fc20 [term 1 FOLLOWER]: Refusing update from remote peer eb8f9ba051bd43b1abce006d158819ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:28:43.614774 30819 consensus_queue.cc:1035] T d8f18b93328f44a39e9431ecf696146e P eb8f9ba051bd43b1abce006d158819ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c03d7c898a8444ab6fd9e89ea1f033d" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 34715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:28:43.614974 30821 consensus_queue.cc:1035] T d8f18b93328f44a39e9431ecf696146e P eb8f9ba051bd43b1abce006d158819ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf38d61408bd4eada8f8ac705a91fc20" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 35415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:28:43.625831 30826 mvcc.cc:205] Tried to move back new op lower bound from 7332595808714854400 to 7332595808619016192. Current Snapshot: MvccSnapshot[applied={T|T < 7332595808714854400}]
I20260923 17:28:43.633337 30333 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:28:43.633533 30333 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:28:43.636044 30333 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3a5d4512441f4aa6a19a564d56352e88: Sending DeleteTablet for 3 replicas of tablet d8f18b93328f44a39e9431ecf696146e
I20260923 17:28:43.636973 30594 tablet_service.cc:1514] Processing DeleteTablet for tablet d8f18b93328f44a39e9431ecf696146e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:43 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:60450
I20260923 17:28:43.637164 30728 tablet_service.cc:1514] Processing DeleteTablet for tablet d8f18b93328f44a39e9431ecf696146e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:43 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:53764
I20260923 17:28:43.637571 30460 tablet_service.cc:1514] Processing DeleteTablet for tablet d8f18b93328f44a39e9431ecf696146e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:43 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:36720
I20260923 17:28:43.637748 30835 tablet_replica.cc:330] T d8f18b93328f44a39e9431ecf696146e P eb8f9ba051bd43b1abce006d158819ea: stopping tablet replica
I20260923 17:28:43.638055 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 30397
I20260923 17:28:43.639345 30835 raft_consensus.cc:2229] T d8f18b93328f44a39e9431ecf696146e P eb8f9ba051bd43b1abce006d158819ea [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:28:43.639880 30835 raft_consensus.cc:2258] T d8f18b93328f44a39e9431ecf696146e P eb8f9ba051bd43b1abce006d158819ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:43.640196 30834 tablet_replica.cc:330] T d8f18b93328f44a39e9431ecf696146e P cf38d61408bd4eada8f8ac705a91fc20: stopping tablet replica
I20260923 17:28:43.640576 30834 raft_consensus.cc:2229] T d8f18b93328f44a39e9431ecf696146e P cf38d61408bd4eada8f8ac705a91fc20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:28:43.640906 30834 raft_consensus.cc:2258] T d8f18b93328f44a39e9431ecf696146e P cf38d61408bd4eada8f8ac705a91fc20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:43.641444 30835 ts_tablet_manager.cc:1905] T d8f18b93328f44a39e9431ecf696146e P eb8f9ba051bd43b1abce006d158819ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:43.642740 30834 ts_tablet_manager.cc:1905] T d8f18b93328f44a39e9431ecf696146e P cf38d61408bd4eada8f8ac705a91fc20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:43.644239 30835 ts_tablet_manager.cc:1918] T d8f18b93328f44a39e9431ecf696146e P eb8f9ba051bd43b1abce006d158819ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:28:43.644337 30835 log.cc:1196] T d8f18b93328f44a39e9431ecf696146e P eb8f9ba051bd43b1abce006d158819ea: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-2/wal/wals/d8f18b93328f44a39e9431ecf696146e
I20260923 17:28:43.644666 30835 ts_tablet_manager.cc:1939] T d8f18b93328f44a39e9431ecf696146e P eb8f9ba051bd43b1abce006d158819ea: Deleting consensus metadata
I20260923 17:28:43.646065 30319 catalog_manager.cc:4841] TS eb8f9ba051bd43b1abce006d158819ea (127.11.229.3:35461): tablet d8f18b93328f44a39e9431ecf696146e (table test-table [id=0ffd02549b63406188e41de7cc120137]) successfully deleted
I20260923 17:28:43.646231 30834 ts_tablet_manager.cc:1918] T d8f18b93328f44a39e9431ecf696146e P cf38d61408bd4eada8f8ac705a91fc20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:28:43.646351 30834 log.cc:1196] T d8f18b93328f44a39e9431ecf696146e P cf38d61408bd4eada8f8ac705a91fc20: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184271597484-12180-0/minicluster-data/ts-1/wal/wals/d8f18b93328f44a39e9431ecf696146e
I20260923 17:28:43.646694 30834 ts_tablet_manager.cc:1939] T d8f18b93328f44a39e9431ecf696146e P cf38d61408bd4eada8f8ac705a91fc20: Deleting consensus metadata
W20260923 17:28:43.647568 30320 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.229.1:34715 (error 108)
I20260923 17:28:43.647802 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 30531
I20260923 17:28:43.648090 30319 catalog_manager.cc:4841] TS cf38d61408bd4eada8f8ac705a91fc20 (127.11.229.2:35415): tablet d8f18b93328f44a39e9431ecf696146e (table test-table [id=0ffd02549b63406188e41de7cc120137]) successfully deleted
I20260923 17:28:43.649085 30805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:43.648720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:34715 (local address 127.0.0.1:36732)
0923 17:28:43.648892 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:43.648976 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.1:34715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260923 17:28:43.649289 30320 catalog_manager.cc:4584] TS 0c03d7c898a8444ab6fd9e89ea1f033d (127.11.229.1:34715): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d8f18b93328f44a39e9431ecf696146e: Network error: Client connection negotiation failed: client connection to 127.11.229.1:34715: connect: Connection refused (error 111)
I20260923 17:28:43.655325 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 30665
I20260923 17:28:43.661099 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 30303
2026-09-23T17:28:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2492 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-23T17:28:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:28:43Z Disabled control of system clock
I20260923 17:28:43.693384 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:35229
--webserver_interface=127.11.229.62
--webserver_port=0
--builtin_ntp_servers=127.11.229.20:37803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:35229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 17:28:43.806588 30841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:43.806886 30841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:43.806926 30841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:43.811481 30841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:28:43.811543 30841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:43.811559 30841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:43.811578 30841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:28:43.811594 30841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:28:43.816507 30841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:37803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:35229
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:35229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:43.817718 30841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:43.819092 30841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:43.827775 30849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:43.827824 30847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:43.827800 30846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:43.828007 30841 server_base.cc:1034] running on GCE node
I20260923 17:28:43.828684 30841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:43.829907 30841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:43.831105 30841 hybrid_clock.cc:648] HybridClock initialized: now 1790184523831092 us; error 30 us; skew 500 ppm
I20260923 17:28:43.833181 30841 webserver.cc:460] Webserver started at http://127.11.229.62:34133/ using document root <none> and password file <none>
I20260923 17:28:43.833832 30841 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:43.833935 30841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:43.834167 30841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:43.836467 30841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "8a901ab7a07a43d190e7ccb1089e7f07"
format_stamp: "Formatted at 2026-09-23 17:28:43 on dist-test-slave-ptl9"
server_key: "0e3df9aa29defb57bfd44bbfa99b49b8"
server_key_iv: "96fbd511753c4ed6eb1fc8d7f82bde47"
server_key_version: "encryptionkey@0"
I20260923 17:28:43.837219 30841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "8a901ab7a07a43d190e7ccb1089e7f07"
format_stamp: "Formatted at 2026-09-23 17:28:43 on dist-test-slave-ptl9"
server_key: "0e3df9aa29defb57bfd44bbfa99b49b8"
server_key_iv: "96fbd511753c4ed6eb1fc8d7f82bde47"
server_key_version: "encryptionkey@0"
I20260923 17:28:43.841530 30841 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:28:43.844275 30855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:43.845755 30841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:28:43.845909 30841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "8a901ab7a07a43d190e7ccb1089e7f07"
format_stamp: "Formatted at 2026-09-23 17:28:43 on dist-test-slave-ptl9"
server_key: "0e3df9aa29defb57bfd44bbfa99b49b8"
server_key_iv: "96fbd511753c4ed6eb1fc8d7f82bde47"
server_key_version: "encryptionkey@0"
I20260923 17:28:43.846035 30841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:43.862636 30841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:43.863420 30841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:43.863639 30841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:43.870642 30841 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:35229
I20260923 17:28:43.870671 30907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:35229 every 8 connection(s)
I20260923 17:28:43.871588 30841 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:28:43.874809 30908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:43.879560 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 30841
I20260923 17:28:43.879722 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:28:43.880020 12180 external_mini_cluster.cc:1428] Setting key 2417d38003f4d17d95fe619583b16392
I20260923 17:28:43.881196 30908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a901ab7a07a43d190e7ccb1089e7f07: Bootstrap starting.
I20260923 17:28:43.883827 30908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a901ab7a07a43d190e7ccb1089e7f07: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:43.884833 30908 log.cc:824] T 00000000000000000000000000000000 P 8a901ab7a07a43d190e7ccb1089e7f07: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:43.886933 30908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a901ab7a07a43d190e7ccb1089e7f07: No bootstrap required, opened a new log
I20260923 17:28:43.890096 30911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:43.881588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42304 (local address 127.11.229.62:35229)
0923 17:28:43.882285 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:43.882302 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:43.882339 (+ 37us) server_negotiation.cc:407] Connection header received
0923 17:28:43.883047 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:43.883067 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:43.883514 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:43.883880 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:43.884377 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.885647 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.886594 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.887148 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.887332 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:43.888208 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:43.888214 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:43.888229 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:43.888644 (+ 415us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:43.888702 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:43.888722 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:43.888834 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:43.888976 (+ 142us) server_negotiation.cc:299] Negotiation successful
0923 17:28:43.889302 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":139,"threads_started":1}
I20260923 17:28:43.890857 30908 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a901ab7a07a43d190e7ccb1089e7f07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a901ab7a07a43d190e7ccb1089e7f07" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 35229 } }
I20260923 17:28:43.891145 30908 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a901ab7a07a43d190e7ccb1089e7f07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:43.891194 30908 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a901ab7a07a43d190e7ccb1089e7f07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a901ab7a07a43d190e7ccb1089e7f07, State: Initialized, Role: FOLLOWER
I20260923 17:28:43.891673 30908 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a901ab7a07a43d190e7ccb1089e7f07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a901ab7a07a43d190e7ccb1089e7f07" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 35229 } }
I20260923 17:28:43.891848 30908 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a901ab7a07a43d190e7ccb1089e7f07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:28:43.891916 30908 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a901ab7a07a43d190e7ccb1089e7f07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:28:43.892010 30908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a901ab7a07a43d190e7ccb1089e7f07 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:43.893147 30908 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a901ab7a07a43d190e7ccb1089e7f07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a901ab7a07a43d190e7ccb1089e7f07" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 35229 } }
I20260923 17:28:43.893537 30908 leader_election.cc:302] T 00000000000000000000000000000000 P 8a901ab7a07a43d190e7ccb1089e7f07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a901ab7a07a43d190e7ccb1089e7f07; no voters:
I20260923 17:28:43.893813 30908 leader_election.cc:288] T 00000000000000000000000000000000 P 8a901ab7a07a43d190e7ccb1089e7f07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:28:43.893966 30913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a901ab7a07a43d190e7ccb1089e7f07 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:43.894213 30913 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a901ab7a07a43d190e7ccb1089e7f07 [term 1 LEADER]: Becoming Leader. State: Replica: 8a901ab7a07a43d190e7ccb1089e7f07, State: Running, Role: LEADER
I20260923 17:28:43.894560 30913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a901ab7a07a43d190e7ccb1089e7f07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a901ab7a07a43d190e7ccb1089e7f07" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 35229 } }
I20260923 17:28:43.895077 30908 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a901ab7a07a43d190e7ccb1089e7f07 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:28:43.896308 30914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a901ab7a07a43d190e7ccb1089e7f07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a901ab7a07a43d190e7ccb1089e7f07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a901ab7a07a43d190e7ccb1089e7f07" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 35229 } } }
I20260923 17:28:43.896438 30914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a901ab7a07a43d190e7ccb1089e7f07 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:43.896292 30915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a901ab7a07a43d190e7ccb1089e7f07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a901ab7a07a43d190e7ccb1089e7f07. Latest consensus state: current_term: 1 leader_uuid: "8a901ab7a07a43d190e7ccb1089e7f07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a901ab7a07a43d190e7ccb1089e7f07" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 35229 } } }
I20260923 17:28:43.896607 30915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a901ab7a07a43d190e7ccb1089e7f07 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:43.897018 30922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:28:43.900550 30922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:28:43.906955 30922 catalog_manager.cc:1347] Generated new cluster ID: 27ab0058afd84709886dbdd94a408c8e
I20260923 17:28:43.907035 30922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:28:43.916230 30922 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:28:43.917884 30922 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:28:43.925346 30922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a901ab7a07a43d190e7ccb1089e7f07: Generated new TSK 0
I20260923 17:28:43.926335 30922 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:28:43.933766 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.1:0
--local_ip_for_outbound_sockets=127.11.229.1
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:35229
--builtin_ntp_servers=127.11.229.20:37803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 17:28:44.052937 30932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:44.053248 30932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:44.053386 30932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:44.057528 30932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:44.057617 30932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:44.057708 30932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.1
I20260923 17:28:44.062579 30932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:37803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:35229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.1
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:44.063740 30932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:44.065193 30932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:44.074883 30938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:44.074913 30937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:44.075053 30932 server_base.cc:1034] running on GCE node
W20260923 17:28:44.074926 30940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:44.075635 30932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:44.076331 30932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:44.077497 30932 hybrid_clock.cc:648] HybridClock initialized: now 1790184524077480 us; error 28 us; skew 500 ppm
I20260923 17:28:44.079453 30932 webserver.cc:460] Webserver started at http://127.11.229.1:34821/ using document root <none> and password file <none>
I20260923 17:28:44.080154 30932 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:44.080229 30932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:44.080462 30932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:44.082643 30932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-0/data/instance:
uuid: "2c43962ab5014cbf88e3ac5d558e7ba8"
format_stamp: "Formatted at 2026-09-23 17:28:44 on dist-test-slave-ptl9"
server_key: "90cedb1d57d4c42a57dbd9bd3f008b3b"
server_key_iv: "23030a9be41f19094a8e6aba933e9d08"
server_key_version: "encryptionkey@0"
I20260923 17:28:44.083242 30932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance:
uuid: "2c43962ab5014cbf88e3ac5d558e7ba8"
format_stamp: "Formatted at 2026-09-23 17:28:44 on dist-test-slave-ptl9"
server_key: "90cedb1d57d4c42a57dbd9bd3f008b3b"
server_key_iv: "23030a9be41f19094a8e6aba933e9d08"
server_key_version: "encryptionkey@0"
I20260923 17:28:44.087114 30932 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:28:44.089632 30946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:44.090991 30932 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 17:28:44.091142 30932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-0/wal
uuid: "2c43962ab5014cbf88e3ac5d558e7ba8"
format_stamp: "Formatted at 2026-09-23 17:28:44 on dist-test-slave-ptl9"
server_key: "90cedb1d57d4c42a57dbd9bd3f008b3b"
server_key_iv: "23030a9be41f19094a8e6aba933e9d08"
server_key_version: "encryptionkey@0"
I20260923 17:28:44.091261 30932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:44.135001 30932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:44.135763 30932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:44.135973 30932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:44.136636 30932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:44.137805 30932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:44.137873 30932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:44.137949 30932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:44.138002 30932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:44.147135 30932 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.1:38387
I20260923 17:28:44.147248 31059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.1:38387 every 8 connection(s)
I20260923 17:28:44.148167 30932 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
I20260923 17:28:44.151185 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 30932
I20260923 17:28:44.151335 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance
I20260923 17:28:44.151660 12180 external_mini_cluster.cc:1428] Setting key bae4f1377dfeee007df1f397152aa111
I20260923 17:28:44.154006 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.2:0
--local_ip_for_outbound_sockets=127.11.229.2
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:35229
--builtin_ntp_servers=127.11.229.20:37803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 17:28:44.158789 30911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:44.150302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:56495 (local address 127.11.229.62:35229)
0923 17:28:44.150473 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:44.150477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:44.151742 (+ 1265us) server_negotiation.cc:407] Connection header received
0923 17:28:44.153612 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:44.153615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:44.153668 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:44.153771 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:44.154799 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:44.155343 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:44.156527 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:44.156767 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:44.156881 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:44.158208 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:44.158212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:44.158214 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:44.158228 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:44.158249 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:44.158257 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:44.158375 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:44.158553 (+ 178us) server_negotiation.cc:299] Negotiation successful
0923 17:28:44.158625 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 17:28:44.159785 31062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:44.150755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:35229 (local address 127.11.229.1:56495)
0923 17:28:44.151567 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:44.151610 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:44.153346 (+ 1736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:44.153963 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:44.153971 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:44.154079 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:44.154644 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:44.154658 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:44.155474 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:44.155478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:44.156394 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:44.156404 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:44.156930 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:44.156934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:44.157113 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:44.157889 (+ 776us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:28:44.157891 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:44.157938 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:28:44.158015 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:44.158378 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:44.158392 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:44.158400 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 17:28:44.158542 (+ 142us) client_negotiation.cc:240] Negotiation successful
0923 17:28:44.158886 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":174,"threads_started":1}
I20260923 17:28:44.161549 31060 heartbeater.cc:347] Connected to a master server at 127.11.229.62:35229
I20260923 17:28:44.161821 31060 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:44.162431 31060 heartbeater.cc:511] Master 127.11.229.62:35229 requested a full tablet report, sending...
I20260923 17:28:44.164378 30872 ts_manager.cc:194] Registered new tserver with Master: 2c43962ab5014cbf88e3ac5d558e7ba8 (127.11.229.1:38387)
I20260923 17:28:44.166157 30872 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.229.1:56495
W20260923 17:28:44.280355 31063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:44.280653 31063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:44.280694 31063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:44.284900 31063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:44.284970 31063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:44.285029 31063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.2
I20260923 17:28:44.290110 31063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:37803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:35229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.2
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:44.291361 31063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:44.292886 31063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:44.303356 31068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:44.303385 31071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:44.303347 31069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:44.303633 31063 server_base.cc:1034] running on GCE node
I20260923 17:28:44.304160 31063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:44.304803 31063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:44.305958 31063 hybrid_clock.cc:648] HybridClock initialized: now 1790184524305925 us; error 45 us; skew 500 ppm
I20260923 17:28:44.307749 31063 webserver.cc:460] Webserver started at http://127.11.229.2:42013/ using document root <none> and password file <none>
I20260923 17:28:44.308321 31063 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:44.308383 31063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:44.308627 31063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:44.310797 31063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-1/data/instance:
uuid: "9ca2ff4b38804d0291b810f833b5ad01"
format_stamp: "Formatted at 2026-09-23 17:28:44 on dist-test-slave-ptl9"
server_key: "4e4f0506426afad727041073cfc79189"
server_key_iv: "35f414fe23ccebaf1b128c1b5d863e70"
server_key_version: "encryptionkey@0"
I20260923 17:28:44.311367 31063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance:
uuid: "9ca2ff4b38804d0291b810f833b5ad01"
format_stamp: "Formatted at 2026-09-23 17:28:44 on dist-test-slave-ptl9"
server_key: "4e4f0506426afad727041073cfc79189"
server_key_iv: "35f414fe23ccebaf1b128c1b5d863e70"
server_key_version: "encryptionkey@0"
I20260923 17:28:44.315254 31063 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:28:44.317984 31077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:44.319720 31063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:28:44.319844 31063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-1/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-1/wal
uuid: "9ca2ff4b38804d0291b810f833b5ad01"
format_stamp: "Formatted at 2026-09-23 17:28:44 on dist-test-slave-ptl9"
server_key: "4e4f0506426afad727041073cfc79189"
server_key_iv: "35f414fe23ccebaf1b128c1b5d863e70"
server_key_version: "encryptionkey@0"
I20260923 17:28:44.320011 31063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:44.346266 31063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:44.347020 31063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:44.347221 31063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:44.347846 31063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:44.348821 31063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:44.348868 31063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:44.348908 31063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:44.348963 31063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:44.358511 31063 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.2:40135
I20260923 17:28:44.358533 31190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.2:40135 every 8 connection(s)
I20260923 17:28:44.359475 31063 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
I20260923 17:28:44.361900 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 31063
I20260923 17:28:44.362047 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance
I20260923 17:28:44.362387 12180 external_mini_cluster.cc:1428] Setting key 64652f2c6840d0fd0d2e3a59e5edbba3
I20260923 17:28:44.364662 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.3:0
--local_ip_for_outbound_sockets=127.11.229.3
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:35229
--builtin_ntp_servers=127.11.229.20:37803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 17:28:44.370250 30911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:44.361654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:35355 (local address 127.11.229.62:35229)
0923 17:28:44.361820 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:44.361824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:44.363807 (+ 1983us) server_negotiation.cc:407] Connection header received
0923 17:28:44.365095 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:44.365098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:44.365160 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:44.365245 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:44.366375 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:44.366918 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:44.368122 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:44.368361 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:44.368470 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:44.369735 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:44.369739 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:44.369742 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:44.369762 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:44.369781 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:44.369789 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:44.369865 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:44.370028 (+ 163us) server_negotiation.cc:299] Negotiation successful
0923 17:28:44.370096 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 17:28:44.371196 31193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:44.362249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:35229 (local address 127.11.229.2:35355)
0923 17:28:44.363102 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:44.363148 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:44.364857 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:44.365469 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:44.365477 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:44.365573 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:44.366217 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:44.366234 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:44.367060 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:44.367062 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:44.367986 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:44.367997 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:44.368537 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:44.368543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:44.368715 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:44.369450 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:28:44.369452 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:44.369497 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:28:44.369571 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:44.369879 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:44.369889 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:44.369900 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 17:28:44.370027 (+ 127us) client_negotiation.cc:240] Negotiation successful
0923 17:28:44.370344 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":145,"threads_started":1}
I20260923 17:28:44.372426 31191 heartbeater.cc:347] Connected to a master server at 127.11.229.62:35229
I20260923 17:28:44.372714 31191 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:44.373505 31191 heartbeater.cc:511] Master 127.11.229.62:35229 requested a full tablet report, sending...
I20260923 17:28:44.374804 30872 ts_manager.cc:194] Registered new tserver with Master: 9ca2ff4b38804d0291b810f833b5ad01 (127.11.229.2:40135)
I20260923 17:28:44.375669 30872 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.229.2:35355
W20260923 17:28:44.484645 31194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:44.484920 31194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:44.484961 31194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:44.488902 31194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:44.488961 31194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:44.489017 31194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.3
I20260923 17:28:44.493809 31194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:37803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:35229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.3
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:44.495023 31194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:44.496497 31194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 17:28:44.506224 31194 server_base.cc:1034] running on GCE node
W20260923 17:28:44.506168 31202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:44.506165 31199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:44.506162 31200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:44.506897 31194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:44.507578 31194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:44.508759 31194 hybrid_clock.cc:648] HybridClock initialized: now 1790184524508712 us; error 57 us; skew 500 ppm
I20260923 17:28:44.510653 31194 webserver.cc:460] Webserver started at http://127.11.229.3:46463/ using document root <none> and password file <none>
I20260923 17:28:44.511240 31194 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:44.511329 31194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:44.511557 31194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:44.513833 31194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-2/data/instance:
uuid: "f5380a674ad244a9a4d4002204a754c1"
format_stamp: "Formatted at 2026-09-23 17:28:44 on dist-test-slave-ptl9"
server_key: "adcf2f2a357c3ad09f73f41fa92c117d"
server_key_iv: "93bccfd57948203379a2e899c6cb8010"
server_key_version: "encryptionkey@0"
I20260923 17:28:44.514448 31194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance:
uuid: "f5380a674ad244a9a4d4002204a754c1"
format_stamp: "Formatted at 2026-09-23 17:28:44 on dist-test-slave-ptl9"
server_key: "adcf2f2a357c3ad09f73f41fa92c117d"
server_key_iv: "93bccfd57948203379a2e899c6cb8010"
server_key_version: "encryptionkey@0"
I20260923 17:28:44.518555 31194 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260923 17:28:44.521446 31208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:44.522853 31194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:28:44.523017 31194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-2/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-2/wal
uuid: "f5380a674ad244a9a4d4002204a754c1"
format_stamp: "Formatted at 2026-09-23 17:28:44 on dist-test-slave-ptl9"
server_key: "adcf2f2a357c3ad09f73f41fa92c117d"
server_key_iv: "93bccfd57948203379a2e899c6cb8010"
server_key_version: "encryptionkey@0"
I20260923 17:28:44.523209 31194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:44.543968 31194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:44.544726 31194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:44.544937 31194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:44.545590 31194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:44.546567 31194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:44.546636 31194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:44.546706 31194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:44.546754 31194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:44.556195 31194 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.3:34023
I20260923 17:28:44.556216 31321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.3:34023 every 8 connection(s)
I20260923 17:28:44.557161 31194 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
I20260923 17:28:44.561529 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 31194
I20260923 17:28:44.561673 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance
I20260923 17:28:44.562005 12180 external_mini_cluster.cc:1428] Setting key 87e505001f5610fab559de3583063b57
I20260923 17:28:44.567452 30911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:44.559214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:52589 (local address 127.11.229.62:35229)
0923 17:28:44.559393 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:44.559399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:44.560449 (+ 1050us) server_negotiation.cc:407] Connection header received
0923 17:28:44.562039 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:44.562045 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:44.562107 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:44.562209 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:44.563341 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:44.564092 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:44.565343 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:44.565574 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:44.565690 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:44.566907 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:44.566911 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:44.566913 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:44.566929 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:44.566954 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:44.566962 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:44.567036 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:44.567207 (+ 171us) server_negotiation.cc:299] Negotiation successful
0923 17:28:44.567275 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 17:28:44.568403 31324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:44.559614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:35229 (local address 127.11.229.3:52589)
0923 17:28:44.560299 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:44.560344 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:44.561800 (+ 1456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:44.562373 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:44.562380 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:44.562490 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:44.563190 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:44.563207 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:44.564230 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:44.564233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:44.565137 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:44.565154 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:44.565740 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:44.565743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:44.565925 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:44.566583 (+ 658us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:28:44.566585 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:44.566630 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:28:44.566706 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:44.567050 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:44.567060 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:44.567066 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 17:28:44.567195 (+ 129us) client_negotiation.cc:240] Negotiation successful
0923 17:28:44.567508 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":108,"threads_started":1}
I20260923 17:28:44.569779 31322 heartbeater.cc:347] Connected to a master server at 127.11.229.62:35229
I20260923 17:28:44.570072 31322 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:44.570691 31322 heartbeater.cc:511] Master 127.11.229.62:35229 requested a full tablet report, sending...
I20260923 17:28:44.571853 30872 ts_manager.cc:194] Registered new tserver with Master: f5380a674ad244a9a4d4002204a754c1 (127.11.229.3:34023)
I20260923 17:28:44.572472 30872 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.229.3:52589
I20260923 17:28:44.576097 12180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:28:44.582026 30911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:44.578111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46004 (local address 127.11.229.62:35229)
0923 17:28:44.578252 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:44.578265 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:44.578428 (+ 163us) server_negotiation.cc:407] Connection header received
0923 17:28:44.578541 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:44.578543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:44.578583 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:44.578652 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:44.578949 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:44.579471 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:44.580302 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:44.580517 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:44.580592 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:44.581517 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:44.581520 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:44.581522 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:44.581534 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:44.581551 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:44.581559 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:44.581646 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:44.581785 (+ 139us) server_negotiation.cc:299] Negotiation successful
0923 17:28:44.581840 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 17:28:44.585141 30872 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:28:44.588177 30872 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:28:44.602946 31333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:44.597114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:34023 (local address 127.0.0.1:56408)
0923 17:28:44.597663 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:44.597715 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:44.597875 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:44.598442 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:44.598448 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:44.598476 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:44.598643 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:44.598652 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:44.600078 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:44.600081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:44.601445 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:44.601455 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:44.602501 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:44.602504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:44.602597 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:44.602629 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 17:28:44.602705 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 17:28:44.602801 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":1,"thread_start_us":111,"threads_started":1}
I20260923 17:28:44.603029 31334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:44.597299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56408 (local address 127.11.229.3:34023)
0923 17:28:44.597627 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:44.597640 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:44.597802 (+ 162us) server_negotiation.cc:407] Connection header received
0923 17:28:44.598035 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:44.598041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:44.598261 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:44.598829 (+ 568us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:44.598979 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:44.599958 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:44.601584 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:44.602378 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:44.602487 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:44.602688 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:44.602743 (+ 55us) server_negotiation.cc:299] Negotiation successful
0923 17:28:44.602849 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":109,"threads_started":1}
I20260923 17:28:44.603883 31335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:44.597901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:38387 (local address 127.0.0.1:55592)
0923 17:28:44.598134 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:44.598147 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:44.598219 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:44.599329 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:44.599332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:44.599352 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:44.599435 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:44.599441 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:44.600645 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:44.600649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:44.602077 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:44.602086 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:44.603495 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:44.603498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:44.603566 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:44.603585 (+ 19us) client_negotiation.cc:769] Sending connection context
0923 17:28:44.603676 (+ 91us) client_negotiation.cc:240] Negotiation successful
0923 17:28:44.603749 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":140,"thread_start_us":96,"threads_started":1}
I20260923 17:28:44.603989 31336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:44.598070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55592 (local address 127.11.229.1:38387)
0923 17:28:44.598866 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:44.598873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:44.598905 (+ 32us) server_negotiation.cc:407] Connection header received
0923 17:28:44.598983 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:44.598989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:44.599172 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:44.599333 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:44.599539 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:44.600508 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:44.602218 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:44.602997 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:44.603112 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:44.603324 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:44.603712 (+ 388us) server_negotiation.cc:299] Negotiation successful
0923 17:28:44.603810 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":89,"threads_started":1}
I20260923 17:28:44.604910 31337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:44.598214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:40135 (local address 127.0.0.1:57238)
0923 17:28:44.598582 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:44.598598 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:44.598677 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:44.599539 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:44.599541 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:44.599560 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:44.599642 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:44.599648 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:44.600711 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:44.600718 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:44.602074 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:44.602086 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:44.604523 (+ 2437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:44.604527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:44.604614 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:44.604635 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 17:28:44.604707 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 17:28:44.604776 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":63,"thread_start_us":107,"threads_started":1}
I20260923 17:28:44.605525 31338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:44.598523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57238 (local address 127.11.229.2:40135)
0923 17:28:44.599022 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:44.599033 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:44.599051 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:28:44.599113 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:44.599117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:44.599267 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:44.599400 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:44.599764 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:44.600599 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:44.603393 (+ 2794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:44.604157 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:44.604262 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:44.604457 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:44.605246 (+ 789us) server_negotiation.cc:299] Negotiation successful
0923 17:28:44.605377 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":110,"threads_started":1}
I20260923 17:28:44.606387 31256 tablet_service.cc:1467] Processing CreateTablet for tablet 4424289e5aed448e8f8658d7437027e0 (DEFAULT_TABLE table=test-table [id=ebcc060439c94167a0f7eae5f98c4284]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:44.606874 30994 tablet_service.cc:1467] Processing CreateTablet for tablet 4424289e5aed448e8f8658d7437027e0 (DEFAULT_TABLE table=test-table [id=ebcc060439c94167a0f7eae5f98c4284]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:44.607625 31256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4424289e5aed448e8f8658d7437027e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:44.607788 30994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4424289e5aed448e8f8658d7437027e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:44.608295 31125 tablet_service.cc:1467] Processing CreateTablet for tablet 4424289e5aed448e8f8658d7437027e0 (DEFAULT_TABLE table=test-table [id=ebcc060439c94167a0f7eae5f98c4284]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:44.609253 31125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4424289e5aed448e8f8658d7437027e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:44.614527 31339 tablet_bootstrap.cc:492] T 4424289e5aed448e8f8658d7437027e0 P 2c43962ab5014cbf88e3ac5d558e7ba8: Bootstrap starting.
I20260923 17:28:44.616045 31341 tablet_bootstrap.cc:492] T 4424289e5aed448e8f8658d7437027e0 P 9ca2ff4b38804d0291b810f833b5ad01: Bootstrap starting.
I20260923 17:28:44.616803 31339 tablet_bootstrap.cc:654] T 4424289e5aed448e8f8658d7437027e0 P 2c43962ab5014cbf88e3ac5d558e7ba8: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:44.617080 31340 tablet_bootstrap.cc:492] T 4424289e5aed448e8f8658d7437027e0 P f5380a674ad244a9a4d4002204a754c1: Bootstrap starting.
I20260923 17:28:44.617660 31339 log.cc:824] T 4424289e5aed448e8f8658d7437027e0 P 2c43962ab5014cbf88e3ac5d558e7ba8: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:44.618508 31341 tablet_bootstrap.cc:654] T 4424289e5aed448e8f8658d7437027e0 P 9ca2ff4b38804d0291b810f833b5ad01: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:44.619192 31340 tablet_bootstrap.cc:654] T 4424289e5aed448e8f8658d7437027e0 P f5380a674ad244a9a4d4002204a754c1: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:44.619354 31341 log.cc:824] T 4424289e5aed448e8f8658d7437027e0 P 9ca2ff4b38804d0291b810f833b5ad01: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:44.619752 31339 tablet_bootstrap.cc:492] T 4424289e5aed448e8f8658d7437027e0 P 2c43962ab5014cbf88e3ac5d558e7ba8: No bootstrap required, opened a new log
I20260923 17:28:44.620009 31339 ts_tablet_manager.cc:1397] T 4424289e5aed448e8f8658d7437027e0 P 2c43962ab5014cbf88e3ac5d558e7ba8: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260923 17:28:44.620088 31340 log.cc:824] T 4424289e5aed448e8f8658d7437027e0 P f5380a674ad244a9a4d4002204a754c1: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:44.621485 31341 tablet_bootstrap.cc:492] T 4424289e5aed448e8f8658d7437027e0 P 9ca2ff4b38804d0291b810f833b5ad01: No bootstrap required, opened a new log
I20260923 17:28:44.621697 31341 ts_tablet_manager.cc:1397] T 4424289e5aed448e8f8658d7437027e0 P 9ca2ff4b38804d0291b810f833b5ad01: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260923 17:28:44.622231 31340 tablet_bootstrap.cc:492] T 4424289e5aed448e8f8658d7437027e0 P f5380a674ad244a9a4d4002204a754c1: No bootstrap required, opened a new log
I20260923 17:28:44.622421 31340 ts_tablet_manager.cc:1397] T 4424289e5aed448e8f8658d7437027e0 P f5380a674ad244a9a4d4002204a754c1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 17:28:44.623883 31339 raft_consensus.cc:348] T 4424289e5aed448e8f8658d7437027e0 P 2c43962ab5014cbf88e3ac5d558e7ba8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5380a674ad244a9a4d4002204a754c1" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34023 } } peers { permanent_uuid: "2c43962ab5014cbf88e3ac5d558e7ba8" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 38387 } } peers { permanent_uuid: "9ca2ff4b38804d0291b810f833b5ad01" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 40135 } }
I20260923 17:28:44.624130 31339 raft_consensus.cc:374] T 4424289e5aed448e8f8658d7437027e0 P 2c43962ab5014cbf88e3ac5d558e7ba8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:44.624214 31339 raft_consensus.cc:729] T 4424289e5aed448e8f8658d7437027e0 P 2c43962ab5014cbf88e3ac5d558e7ba8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c43962ab5014cbf88e3ac5d558e7ba8, State: Initialized, Role: FOLLOWER
I20260923 17:28:44.624750 31339 consensus_queue.cc:260] T 4424289e5aed448e8f8658d7437027e0 P 2c43962ab5014cbf88e3ac5d558e7ba8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5380a674ad244a9a4d4002204a754c1" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34023 } } peers { permanent_uuid: "2c43962ab5014cbf88e3ac5d558e7ba8" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 38387 } } peers { permanent_uuid: "9ca2ff4b38804d0291b810f833b5ad01" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 40135 } }
I20260923 17:28:44.625486 31060 heartbeater.cc:503] Master 127.11.229.62:35229 was elected leader, sending a full tablet report...
I20260923 17:28:44.625864 31339 ts_tablet_manager.cc:1428] T 4424289e5aed448e8f8658d7437027e0 P 2c43962ab5014cbf88e3ac5d558e7ba8: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260923 17:28:44.626338 31341 raft_consensus.cc:348] T 4424289e5aed448e8f8658d7437027e0 P 9ca2ff4b38804d0291b810f833b5ad01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5380a674ad244a9a4d4002204a754c1" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34023 } } peers { permanent_uuid: "2c43962ab5014cbf88e3ac5d558e7ba8" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 38387 } } peers { permanent_uuid: "9ca2ff4b38804d0291b810f833b5ad01" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 40135 } }
I20260923 17:28:44.626526 31340 raft_consensus.cc:348] T 4424289e5aed448e8f8658d7437027e0 P f5380a674ad244a9a4d4002204a754c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5380a674ad244a9a4d4002204a754c1" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34023 } } peers { permanent_uuid: "2c43962ab5014cbf88e3ac5d558e7ba8" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 38387 } } peers { permanent_uuid: "9ca2ff4b38804d0291b810f833b5ad01" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 40135 } }
I20260923 17:28:44.626819 31341 raft_consensus.cc:374] T 4424289e5aed448e8f8658d7437027e0 P 9ca2ff4b38804d0291b810f833b5ad01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:44.626981 31341 raft_consensus.cc:729] T 4424289e5aed448e8f8658d7437027e0 P 9ca2ff4b38804d0291b810f833b5ad01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ca2ff4b38804d0291b810f833b5ad01, State: Initialized, Role: FOLLOWER
I20260923 17:28:44.627471 31341 consensus_queue.cc:260] T 4424289e5aed448e8f8658d7437027e0 P 9ca2ff4b38804d0291b810f833b5ad01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5380a674ad244a9a4d4002204a754c1" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34023 } } peers { permanent_uuid: "2c43962ab5014cbf88e3ac5d558e7ba8" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 38387 } } peers { permanent_uuid: "9ca2ff4b38804d0291b810f833b5ad01" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 40135 } }
I20260923 17:28:44.628155 31191 heartbeater.cc:503] Master 127.11.229.62:35229 was elected leader, sending a full tablet report...
I20260923 17:28:44.628549 31341 ts_tablet_manager.cc:1428] T 4424289e5aed448e8f8658d7437027e0 P 9ca2ff4b38804d0291b810f833b5ad01: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260923 17:28:44.626746 31340 raft_consensus.cc:374] T 4424289e5aed448e8f8658d7437027e0 P f5380a674ad244a9a4d4002204a754c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:44.628685 31340 raft_consensus.cc:729] T 4424289e5aed448e8f8658d7437027e0 P f5380a674ad244a9a4d4002204a754c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5380a674ad244a9a4d4002204a754c1, State: Initialized, Role: FOLLOWER
I20260923 17:28:44.629145 31340 consensus_queue.cc:260] T 4424289e5aed448e8f8658d7437027e0 P f5380a674ad244a9a4d4002204a754c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5380a674ad244a9a4d4002204a754c1" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34023 } } peers { permanent_uuid: "2c43962ab5014cbf88e3ac5d558e7ba8" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 38387 } } peers { permanent_uuid: "9ca2ff4b38804d0291b810f833b5ad01" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 40135 } }
I20260923 17:28:44.630401 31322 heartbeater.cc:503] Master 127.11.229.62:35229 was elected leader, sending a full tablet report...
I20260923 17:28:44.630754 31340 ts_tablet_manager.cc:1428] T 4424289e5aed448e8f8658d7437027e0 P f5380a674ad244a9a4d4002204a754c1: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
W20260923 17:28:44.649464 31061 tablet.cc:2369] T 4424289e5aed448e8f8658d7437027e0 P 2c43962ab5014cbf88e3ac5d558e7ba8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:28:44.676301 31345 raft_consensus.cc:482] T 4424289e5aed448e8f8658d7437027e0 P 2c43962ab5014cbf88e3ac5d558e7ba8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:28:44.676568 31345 raft_consensus.cc:504] T 4424289e5aed448e8f8658d7437027e0 P 2c43962ab5014cbf88e3ac5d558e7ba8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5380a674ad244a9a4d4002204a754c1" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34023 } } peers { permanent_uuid: "2c43962ab5014cbf88e3ac5d558e7ba8" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 38387 } } peers { permanent_uuid: "9ca2ff4b38804d0291b810f833b5ad01" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 40135 } }
I20260923 17:28:44.677847 31345 leader_election.cc:288] T 4424289e5aed448e8f8658d7437027e0 P 2c43962ab5014cbf88e3ac5d558e7ba8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5380a674ad244a9a4d4002204a754c1 (127.11.229.3:34023), 9ca2ff4b38804d0291b810f833b5ad01 (127.11.229.2:40135)
I20260923 17:28:44.683373 31338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:44.678782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:51337 (local address 127.11.229.2:40135)
0923 17:28:44.678917 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:44.678921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:44.679069 (+ 148us) server_negotiation.cc:407] Connection header received
0923 17:28:44.679189 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:44.679192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:44.679234 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:44.679330 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:44.679559 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:44.680282 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:44.681888 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:44.682511 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:44.682639 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:44.682693 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:44.683135 (+ 442us) server_negotiation.cc:299] Negotiation successful
0923 17:28:44.683235 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 17:28:44.683373 31334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:44.678331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:55841 (local address 127.11.229.3:34023)
0923 17:28:44.678484 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:44.678488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:44.678640 (+ 152us) server_negotiation.cc:407] Connection header received
0923 17:28:44.678849 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:44.678855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:44.678910 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:44.678979 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:44.679323 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:44.680099 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:44.681888 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:44.682500 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:44.682640 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:44.682693 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:44.683135 (+ 442us) server_negotiation.cc:299] Negotiation successful
0923 17:28:44.683230 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 17:28:44.683419 31348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:44.678196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:34023 (local address 127.11.229.1:55841)
0923 17:28:44.678527 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:44.678539 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:44.678702 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:44.678999 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:44.679005 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:44.679028 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:44.679204 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:44.679211 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:44.680260 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:44.680264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:44.681723 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:44.681740 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:44.682670 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:44.682674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:44.682898 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:44.682980 (+ 82us) client_negotiation.cc:769] Sending connection context
0923 17:28:44.683081 (+ 101us) client_negotiation.cc:240] Negotiation successful
0923 17:28:44.683228 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":152,"spinlock_wait_cycles":15104,"thread_start_us":120,"threads_started":1}
I20260923 17:28:44.683419 31349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:44.678634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:40135 (local address 127.11.229.1:51337)
0923 17:28:44.678974 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:44.678986 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:44.679090 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:44.679351 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:44.679353 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:44.679368 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:44.679460 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:44.679464 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:44.680373 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:44.680376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:44.681723 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:44.681736 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:44.682669 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:44.682674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:44.682895 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:44.682990 (+ 95us) client_negotiation.cc:769] Sending connection context
0923 17:28:44.683081 (+ 91us) client_negotiation.cc:240] Negotiation successful
0923 17:28:44.683205 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":104,"spinlock_wait_cycles":2816,"thread_start_us":100,"threads_started":1}
I20260923 17:28:44.684265 31145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4424289e5aed448e8f8658d7437027e0" candidate_uuid: "2c43962ab5014cbf88e3ac5d558e7ba8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca2ff4b38804d0291b810f833b5ad01" is_pre_election: true
I20260923 17:28:44.684274 31276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4424289e5aed448e8f8658d7437027e0" candidate_uuid: "2c43962ab5014cbf88e3ac5d558e7ba8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5380a674ad244a9a4d4002204a754c1" is_pre_election: true
I20260923 17:28:44.684639 31276 raft_consensus.cc:2454] T 4424289e5aed448e8f8658d7437027e0 P f5380a674ad244a9a4d4002204a754c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c43962ab5014cbf88e3ac5d558e7ba8 in term 0.
I20260923 17:28:44.684625 31145 raft_consensus.cc:2454] T 4424289e5aed448e8f8658d7437027e0 P 9ca2ff4b38804d0291b810f833b5ad01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c43962ab5014cbf88e3ac5d558e7ba8 in term 0.
I20260923 17:28:44.685187 30950 leader_election.cc:302] T 4424289e5aed448e8f8658d7437027e0 P 2c43962ab5014cbf88e3ac5d558e7ba8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c43962ab5014cbf88e3ac5d558e7ba8, f5380a674ad244a9a4d4002204a754c1; no voters:
I20260923 17:28:44.685514 31345 raft_consensus.cc:2781] T 4424289e5aed448e8f8658d7437027e0 P 2c43962ab5014cbf88e3ac5d558e7ba8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:28:44.685622 31345 raft_consensus.cc:482] T 4424289e5aed448e8f8658d7437027e0 P 2c43962ab5014cbf88e3ac5d558e7ba8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:28:44.685696 31345 raft_consensus.cc:3037] T 4424289e5aed448e8f8658d7437027e0 P 2c43962ab5014cbf88e3ac5d558e7ba8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:44.686827 31345 raft_consensus.cc:504] T 4424289e5aed448e8f8658d7437027e0 P 2c43962ab5014cbf88e3ac5d558e7ba8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5380a674ad244a9a4d4002204a754c1" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34023 } } peers { permanent_uuid: "2c43962ab5014cbf88e3ac5d558e7ba8" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 38387 } } peers { permanent_uuid: "9ca2ff4b38804d0291b810f833b5ad01" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 40135 } }
I20260923 17:28:44.687306 31345 leader_election.cc:288] T 4424289e5aed448e8f8658d7437027e0 P 2c43962ab5014cbf88e3ac5d558e7ba8 [CANDIDATE]: Term 1 election: Requested vote from peers f5380a674ad244a9a4d4002204a754c1 (127.11.229.3:34023), 9ca2ff4b38804d0291b810f833b5ad01 (127.11.229.2:40135)
I20260923 17:28:44.687783 31276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4424289e5aed448e8f8658d7437027e0" candidate_uuid: "2c43962ab5014cbf88e3ac5d558e7ba8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5380a674ad244a9a4d4002204a754c1"
I20260923 17:28:44.687805 31145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4424289e5aed448e8f8658d7437027e0" candidate_uuid: "2c43962ab5014cbf88e3ac5d558e7ba8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca2ff4b38804d0291b810f833b5ad01"
I20260923 17:28:44.687928 31276 raft_consensus.cc:3037] T 4424289e5aed448e8f8658d7437027e0 P f5380a674ad244a9a4d4002204a754c1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:44.687948 31145 raft_consensus.cc:3037] T 4424289e5aed448e8f8658d7437027e0 P 9ca2ff4b38804d0291b810f833b5ad01 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:44.689033 31276 raft_consensus.cc:2454] T 4424289e5aed448e8f8658d7437027e0 P f5380a674ad244a9a4d4002204a754c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c43962ab5014cbf88e3ac5d558e7ba8 in term 1.
I20260923 17:28:44.689564 30950 leader_election.cc:302] T 4424289e5aed448e8f8658d7437027e0 P 2c43962ab5014cbf88e3ac5d558e7ba8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c43962ab5014cbf88e3ac5d558e7ba8, f5380a674ad244a9a4d4002204a754c1; no voters:
I20260923 17:28:44.689703 31145 raft_consensus.cc:2454] T 4424289e5aed448e8f8658d7437027e0 P 9ca2ff4b38804d0291b810f833b5ad01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c43962ab5014cbf88e3ac5d558e7ba8 in term 1.
I20260923 17:28:44.689967 31345 raft_consensus.cc:2781] T 4424289e5aed448e8f8658d7437027e0 P 2c43962ab5014cbf88e3ac5d558e7ba8 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:44.690240 31345 raft_consensus.cc:686] T 4424289e5aed448e8f8658d7437027e0 P 2c43962ab5014cbf88e3ac5d558e7ba8 [term 1 LEADER]: Becoming Leader. State: Replica: 2c43962ab5014cbf88e3ac5d558e7ba8, State: Running, Role: LEADER
I20260923 17:28:44.690644 31345 consensus_queue.cc:237] T 4424289e5aed448e8f8658d7437027e0 P 2c43962ab5014cbf88e3ac5d558e7ba8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5380a674ad244a9a4d4002204a754c1" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34023 } } peers { permanent_uuid: "2c43962ab5014cbf88e3ac5d558e7ba8" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 38387 } } peers { permanent_uuid: "9ca2ff4b38804d0291b810f833b5ad01" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 40135 } }
I20260923 17:28:44.694020 30872 catalog_manager.cc:5495] T 4424289e5aed448e8f8658d7437027e0 P 2c43962ab5014cbf88e3ac5d558e7ba8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c43962ab5014cbf88e3ac5d558e7ba8 (127.11.229.1). New cstate: current_term: 1 leader_uuid: "2c43962ab5014cbf88e3ac5d558e7ba8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5380a674ad244a9a4d4002204a754c1" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c43962ab5014cbf88e3ac5d558e7ba8" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 38387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ca2ff4b38804d0291b810f833b5ad01" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 40135 } health_report { overall_health: UNKNOWN } } }
I20260923 17:28:44.728283 31336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:44.724458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55600 (local address 127.11.229.1:38387)
0923 17:28:44.724579 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:44.724583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:44.724597 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:28:44.724930 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:44.724933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:44.724999 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:44.725105 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:28:44.725395 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:44.725893 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:44.726745 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:44.726956 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:44.727051 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:44.727462 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:28:44.727900 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:28:44.728037 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:44.728132 (+ 95us) server_negotiation.cc:299] Negotiation successful
0923 17:28:44.728167 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 17:28:44.734534 31276 raft_consensus.cc:1261] T 4424289e5aed448e8f8658d7437027e0 P f5380a674ad244a9a4d4002204a754c1 [term 1 FOLLOWER]: Refusing update from remote peer 2c43962ab5014cbf88e3ac5d558e7ba8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:28:44.734534 31145 raft_consensus.cc:1261] T 4424289e5aed448e8f8658d7437027e0 P 9ca2ff4b38804d0291b810f833b5ad01 [term 1 FOLLOWER]: Refusing update from remote peer 2c43962ab5014cbf88e3ac5d558e7ba8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:28:44.735306 31345 consensus_queue.cc:1035] T 4424289e5aed448e8f8658d7437027e0 P 2c43962ab5014cbf88e3ac5d558e7ba8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ca2ff4b38804d0291b810f833b5ad01" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 40135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:28:44.735502 31350 consensus_queue.cc:1035] T 4424289e5aed448e8f8658d7437027e0 P 2c43962ab5014cbf88e3ac5d558e7ba8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5380a674ad244a9a4d4002204a754c1" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 34023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:28:44.752975 30872 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:28:44.753180 30872 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:28:44.755982 30872 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a901ab7a07a43d190e7ccb1089e7f07: Sending DeleteTablet for 3 replicas of tablet 4424289e5aed448e8f8658d7437027e0
I20260923 17:28:44.756856 31256 tablet_service.cc:1514] Processing DeleteTablet for tablet 4424289e5aed448e8f8658d7437027e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:44 UTC) from {username='slave'} at 127.0.0.1:56408
I20260923 17:28:44.756857 30994 tablet_service.cc:1514] Processing DeleteTablet for tablet 4424289e5aed448e8f8658d7437027e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:44 UTC) from {username='slave'} at 127.0.0.1:55592
I20260923 17:28:44.757041 31125 tablet_service.cc:1514] Processing DeleteTablet for tablet 4424289e5aed448e8f8658d7437027e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:44 UTC) from {username='slave'} at 127.0.0.1:57238
I20260923 17:28:44.757418 31363 tablet_replica.cc:330] T 4424289e5aed448e8f8658d7437027e0 P f5380a674ad244a9a4d4002204a754c1: stopping tablet replica
I20260923 17:28:44.757423 31364 tablet_replica.cc:330] T 4424289e5aed448e8f8658d7437027e0 P 2c43962ab5014cbf88e3ac5d558e7ba8: stopping tablet replica
I20260923 17:28:44.757759 31363 raft_consensus.cc:2229] T 4424289e5aed448e8f8658d7437027e0 P f5380a674ad244a9a4d4002204a754c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:28:44.757759 31364 raft_consensus.cc:2229] T 4424289e5aed448e8f8658d7437027e0 P 2c43962ab5014cbf88e3ac5d558e7ba8 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:28:44.758091 31363 raft_consensus.cc:2258] T 4424289e5aed448e8f8658d7437027e0 P f5380a674ad244a9a4d4002204a754c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:44.758138 31364 raft_consensus.cc:2258] T 4424289e5aed448e8f8658d7437027e0 P 2c43962ab5014cbf88e3ac5d558e7ba8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:44.758646 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 30932
I20260923 17:28:44.759333 31365 tablet_replica.cc:330] T 4424289e5aed448e8f8658d7437027e0 P 9ca2ff4b38804d0291b810f833b5ad01: stopping tablet replica
I20260923 17:28:44.759629 31365 raft_consensus.cc:2229] T 4424289e5aed448e8f8658d7437027e0 P 9ca2ff4b38804d0291b810f833b5ad01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:28:44.759908 31365 raft_consensus.cc:2258] T 4424289e5aed448e8f8658d7437027e0 P 9ca2ff4b38804d0291b810f833b5ad01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:44.760862 31363 ts_tablet_manager.cc:1905] T 4424289e5aed448e8f8658d7437027e0 P f5380a674ad244a9a4d4002204a754c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:44.761381 31365 ts_tablet_manager.cc:1905] T 4424289e5aed448e8f8658d7437027e0 P 9ca2ff4b38804d0291b810f833b5ad01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:44.763829 31363 ts_tablet_manager.cc:1918] T 4424289e5aed448e8f8658d7437027e0 P f5380a674ad244a9a4d4002204a754c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:28:44.763953 31363 log.cc:1196] T 4424289e5aed448e8f8658d7437027e0 P f5380a674ad244a9a4d4002204a754c1: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-2/wal/wals/4424289e5aed448e8f8658d7437027e0
I20260923 17:28:44.764072 31365 ts_tablet_manager.cc:1918] T 4424289e5aed448e8f8658d7437027e0 P 9ca2ff4b38804d0291b810f833b5ad01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:28:44.764169 31365 log.cc:1196] T 4424289e5aed448e8f8658d7437027e0 P 9ca2ff4b38804d0291b810f833b5ad01: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184271597484-12180-0/minicluster-data/ts-1/wal/wals/4424289e5aed448e8f8658d7437027e0
I20260923 17:28:44.764286 31363 ts_tablet_manager.cc:1939] T 4424289e5aed448e8f8658d7437027e0 P f5380a674ad244a9a4d4002204a754c1: Deleting consensus metadata
I20260923 17:28:44.764473 31365 ts_tablet_manager.cc:1939] T 4424289e5aed448e8f8658d7437027e0 P 9ca2ff4b38804d0291b810f833b5ad01: Deleting consensus metadata
I20260923 17:28:44.765759 30859 catalog_manager.cc:4841] TS f5380a674ad244a9a4d4002204a754c1 (127.11.229.3:34023): tablet 4424289e5aed448e8f8658d7437027e0 (table test-table [id=ebcc060439c94167a0f7eae5f98c4284]) successfully deleted
I20260923 17:28:44.766027 30859 catalog_manager.cc:4841] TS 9ca2ff4b38804d0291b810f833b5ad01 (127.11.229.2:40135): tablet 4424289e5aed448e8f8658d7437027e0 (table test-table [id=ebcc060439c94167a0f7eae5f98c4284]) successfully deleted
W20260923 17:28:44.766417 30859 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.229.1:38387 (error 108)
I20260923 17:28:44.766621 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 31063
I20260923 17:28:44.767686 31337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:44.767410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:38387 (local address 127.0.0.1:55614)
0923 17:28:44.767543 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:44.767610 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.1:38387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 17:28:44.767863 30859 catalog_manager.cc:4584] TS 2c43962ab5014cbf88e3ac5d558e7ba8 (127.11.229.1:38387): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4424289e5aed448e8f8658d7437027e0: Network error: Client connection negotiation failed: client connection to 127.11.229.1:38387: connect: Connection refused (error 111)
I20260923 17:28:44.774123 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 31194
I20260923 17:28:44.780035 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 30841
2026-09-23T17:28:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1117 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:28:44 dist-test-slave-ptl9 krb5kdc[31369](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:28:44 dist-test-slave-ptl9 krb5kdc[31369](info): set up 2 sockets
Sep 23 17:28:44 dist-test-slave-ptl9 krb5kdc[31369](info): commencing operation
krb5kdc: starting...
W20260923 17:28:45.813552 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:28:45 dist-test-slave-ptl9 krb5kdc[31369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:28:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:28:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.229.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
I20260923 17:28:45.942209 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:44945
--webserver_interface=127.11.229.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:37285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:44945
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:46.055163 31382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:46.055457 31382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:46.055534 31382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:46.059685 31382 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:28:46.059772 31382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:46.059792 31382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:46.059814 31382 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:28:46.059878 31382 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:28:46.065412 31382 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:37285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:44945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:44945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:46.066608 31382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:46.067698 31382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:46.077149 31387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:46.077286 31382 server_base.cc:1034] running on GCE node
W20260923 17:28:46.077088 31390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:46.077169 31388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:46.078091 31382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:46.079305 31382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:46.080510 31382 hybrid_clock.cc:648] HybridClock initialized: now 1790184526080482 us; error 42 us; skew 500 ppm
Sep 23 17:28:46 dist-test-slave-ptl9 krb5kdc[31369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184526, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:46.083451 31382 init.cc:377] Logged in from keytab as kudu/127.11.229.62@KRBTEST.COM (short username kudu)
I20260923 17:28:46.084367 31382 webserver.cc:460] Webserver started at http://127.11.229.62:40987/ using document root <none> and password file <none>
I20260923 17:28:46.084923 31382 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:46.084982 31382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:46.085155 31382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:46.087339 31382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "423705948a254af0acefc05917047ae3"
format_stamp: "Formatted at 2026-09-23 17:28:46 on dist-test-slave-ptl9"
server_key: "fbd736960354ab178bbeded9ff37d29f"
server_key_iv: "bfc4c0c1150556a49eb682a3087b5c18"
server_key_version: "encryptionkey@0"
I20260923 17:28:46.087877 31382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "423705948a254af0acefc05917047ae3"
format_stamp: "Formatted at 2026-09-23 17:28:46 on dist-test-slave-ptl9"
server_key: "fbd736960354ab178bbeded9ff37d29f"
server_key_iv: "bfc4c0c1150556a49eb682a3087b5c18"
server_key_version: "encryptionkey@0"
I20260923 17:28:46.091861 31382 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:28:46.094457 31397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:46.095750 31382 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:28:46.095857 31382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "423705948a254af0acefc05917047ae3"
format_stamp: "Formatted at 2026-09-23 17:28:46 on dist-test-slave-ptl9"
server_key: "fbd736960354ab178bbeded9ff37d29f"
server_key_iv: "bfc4c0c1150556a49eb682a3087b5c18"
server_key_version: "encryptionkey@0"
I20260923 17:28:46.095952 31382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:46.117900 31382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:46.121595 31382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:46.121776 31382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:46.129319 31382 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:44945
I20260923 17:28:46.129374 31449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:44945 every 8 connection(s)
I20260923 17:28:46.130368 31382 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:28:46.133455 31450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:46.139252 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 31382
I20260923 17:28:46.139447 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:28:46.139729 12180 external_mini_cluster.cc:1428] Setting key d1fd1cbc297e813da194f4f3d51df8b5
I20260923 17:28:46.139817 31450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 423705948a254af0acefc05917047ae3: Bootstrap starting.
I20260923 17:28:46.142638 31450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 423705948a254af0acefc05917047ae3: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:46.143554 31450 log.cc:824] T 00000000000000000000000000000000 P 423705948a254af0acefc05917047ae3: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:46.145781 31450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 423705948a254af0acefc05917047ae3: No bootstrap required, opened a new log
Sep 23 17:28:46 dist-test-slave-ptl9 krb5kdc[31369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:28:46.150259 31450 raft_consensus.cc:348] T 00000000000000000000000000000000 P 423705948a254af0acefc05917047ae3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "423705948a254af0acefc05917047ae3" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 44945 } }
I20260923 17:28:46.150513 31450 raft_consensus.cc:374] T 00000000000000000000000000000000 P 423705948a254af0acefc05917047ae3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:46.150573 31450 raft_consensus.cc:729] T 00000000000000000000000000000000 P 423705948a254af0acefc05917047ae3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 423705948a254af0acefc05917047ae3, State: Initialized, Role: FOLLOWER
I20260923 17:28:46.151081 31450 consensus_queue.cc:260] T 00000000000000000000000000000000 P 423705948a254af0acefc05917047ae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "423705948a254af0acefc05917047ae3" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 44945 } }
I20260923 17:28:46.151232 31450 raft_consensus.cc:388] T 00000000000000000000000000000000 P 423705948a254af0acefc05917047ae3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:28:46.151317 31450 raft_consensus.cc:482] T 00000000000000000000000000000000 P 423705948a254af0acefc05917047ae3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:28:46.151418 31450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 423705948a254af0acefc05917047ae3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:46.152333 31450 raft_consensus.cc:504] T 00000000000000000000000000000000 P 423705948a254af0acefc05917047ae3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "423705948a254af0acefc05917047ae3" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 44945 } }
I20260923 17:28:46.152678 31450 leader_election.cc:302] T 00000000000000000000000000000000 P 423705948a254af0acefc05917047ae3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 423705948a254af0acefc05917047ae3; no voters:
I20260923 17:28:46.153024 31450 leader_election.cc:288] T 00000000000000000000000000000000 P 423705948a254af0acefc05917047ae3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:28:46.153115 31455 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 423705948a254af0acefc05917047ae3 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:46.153400 31455 raft_consensus.cc:686] T 00000000000000000000000000000000 P 423705948a254af0acefc05917047ae3 [term 1 LEADER]: Becoming Leader. State: Replica: 423705948a254af0acefc05917047ae3, State: Running, Role: LEADER
I20260923 17:28:46.153774 31455 consensus_queue.cc:237] T 00000000000000000000000000000000 P 423705948a254af0acefc05917047ae3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "423705948a254af0acefc05917047ae3" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 44945 } }
I20260923 17:28:46.154505 31450 sys_catalog.cc:564] T 00000000000000000000000000000000 P 423705948a254af0acefc05917047ae3 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:28:46.154676 31453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:46.141119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58180 (local address 127.11.229.62:44945)
0923 17:28:46.141609 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:46.141626 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:46.141662 (+ 36us) server_negotiation.cc:407] Connection header received
0923 17:28:46.142332 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:46.142370 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:46.142820 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:46.143119 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:46.144033 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:46.145048 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:46.146181 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:46.146644 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:46.146771 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:46.149668 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:46.149674 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:46.149688 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:46.149715 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:46.151525 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:46.151960 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:46.151966 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:46.151973 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:46.152089 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:46.152367 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:46.152372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:46.152374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:46.152865 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:46.153075 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:46.153392 (+ 317us) server_negotiation.cc:299] Negotiation successful
0923 17:28:46.153818 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":200,"threads_started":1}
I20260923 17:28:46.154989 31455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 423705948a254af0acefc05917047ae3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 423705948a254af0acefc05917047ae3. Latest consensus state: current_term: 1 leader_uuid: "423705948a254af0acefc05917047ae3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "423705948a254af0acefc05917047ae3" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 44945 } } }
I20260923 17:28:46.155133 31455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 423705948a254af0acefc05917047ae3 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:46.155368 31455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 423705948a254af0acefc05917047ae3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "423705948a254af0acefc05917047ae3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "423705948a254af0acefc05917047ae3" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 44945 } } }
I20260923 17:28:46.155509 31455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 423705948a254af0acefc05917047ae3 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:46.155916 31460 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:28:46.159523 31460 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:28:46.165723 31460 catalog_manager.cc:1347] Generated new cluster ID: 4671b506987543c38ac95bb49dd9574f
I20260923 17:28:46.165812 31460 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:28:46.203336 31460 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:28:46.204761 31460 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:28:46.209710 31460 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 423705948a254af0acefc05917047ae3: Generated new TSK 0
I20260923 17:28:46.210553 31460 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.229.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab.
Entry for principal kudu/127.11.229.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab.
I20260923 17:28:46.244091 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.1:0
--local_ip_for_outbound_sockets=127.11.229.1
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:44945
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:37285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:46.362219 31476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:46.362522 31476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:46.362569 31476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:46.367096 31476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:46.367195 31476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:46.367353 31476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.1
I20260923 17:28:46.374102 31476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:37285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.1.keytab
--principal=kudu/127.11.229.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:44945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.1
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:46.375496 31476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:46.376626 31476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:46.388550 31482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:46.388540 31484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:46.388540 31481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:46.388921 31476 server_base.cc:1034] running on GCE node
I20260923 17:28:46.389397 31476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:46.390079 31476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:46.391264 31476 hybrid_clock.cc:648] HybridClock initialized: now 1790184526391250 us; error 36 us; skew 500 ppm
Sep 23 17:28:46 dist-test-slave-ptl9 krb5kdc[31369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184526, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:46.394170 31476 init.cc:377] Logged in from keytab as kudu/127.11.229.1@KRBTEST.COM (short username kudu)
I20260923 17:28:46.395061 31476 webserver.cc:460] Webserver started at http://127.11.229.1:33001/ using document root <none> and password file <none>
I20260923 17:28:46.395648 31476 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:46.395736 31476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:46.395968 31476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:46.398185 31476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-0/data/instance:
uuid: "6af5566c43424d399cdb394fad67bca7"
format_stamp: "Formatted at 2026-09-23 17:28:46 on dist-test-slave-ptl9"
server_key: "db52aaca57cceed4bf3ec56e90e44477"
server_key_iv: "f809f4579b35ea467250e2c325f8d88f"
server_key_version: "encryptionkey@0"
I20260923 17:28:46.398758 31476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance:
uuid: "6af5566c43424d399cdb394fad67bca7"
format_stamp: "Formatted at 2026-09-23 17:28:46 on dist-test-slave-ptl9"
server_key: "db52aaca57cceed4bf3ec56e90e44477"
server_key_iv: "f809f4579b35ea467250e2c325f8d88f"
server_key_version: "encryptionkey@0"
I20260923 17:28:46.402750 31476 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:28:46.405537 31491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:46.406837 31476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:28:46.406988 31476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-0/wal
uuid: "6af5566c43424d399cdb394fad67bca7"
format_stamp: "Formatted at 2026-09-23 17:28:46 on dist-test-slave-ptl9"
server_key: "db52aaca57cceed4bf3ec56e90e44477"
server_key_iv: "f809f4579b35ea467250e2c325f8d88f"
server_key_version: "encryptionkey@0"
I20260923 17:28:46.407101 31476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:46.419269 31476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:46.422537 31476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:46.422739 31476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:46.423349 31476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:46.424161 31476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:46.424207 31476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:46.424279 31476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:46.424311 31476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:46.433233 31476 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.1:37023
I20260923 17:28:46.433275 31604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.1:37023 every 8 connection(s)
I20260923 17:28:46.434158 31476 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
I20260923 17:28:46.441956 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 31476
I20260923 17:28:46.442097 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance
I20260923 17:28:46.442415 12180 external_mini_cluster.cc:1428] Setting key f17880e07de6c4fe9514ef44bace6e5d
Sep 23 17:28:46 dist-test-slave-ptl9 krb5kdc[31369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184526, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.1@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:28:46.448302 31453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:46.435987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:48475 (local address 127.11.229.62:44945)
0923 17:28:46.436142 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:46.436147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:46.437017 (+ 870us) server_negotiation.cc:407] Connection header received
0923 17:28:46.438358 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:46.438361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:46.438409 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:46.438483 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:46.439710 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:46.440238 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:46.441381 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:46.441604 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:46.441726 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:46.444354 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:46.444361 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:46.444367 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:46.444400 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:46.446051 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:46.446712 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:46.446715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:46.446716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:46.446789 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:46.447185 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:46.447187 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:46.447188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:46.447362 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:46.447436 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:46.448069 (+ 633us) server_negotiation.cc:299] Negotiation successful
0923 17:28:46.448175 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 17:28:46.449678 31607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:46.436420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:44945 (local address 127.11.229.1:48475)
0923 17:28:46.436864 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:46.436906 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:46.438160 (+ 1254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:46.438646 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:46.438653 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:46.439058 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:46.439573 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:46.439587 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:46.440343 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:46.440345 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:46.441218 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:46.441227 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:46.441770 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:46.441772 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:46.441919 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:46.442578 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:46.442595 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:46.444169 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:46.446172 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:46.446180 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:46.446198 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:46.446612 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:46.446887 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:46.446891 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:46.446893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:46.447085 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:46.447465 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:46.447475 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:46.447816 (+ 341us) client_negotiation.cc:769] Sending connection context
0923 17:28:46.448035 (+ 219us) client_negotiation.cc:240] Negotiation successful
0923 17:28:46.448432 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":117,"threads_started":1}
I20260923 17:28:46.451112 31605 heartbeater.cc:347] Connected to a master server at 127.11.229.62:44945
I20260923 17:28:46.451382 31605 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:46.451930 31605 heartbeater.cc:511] Master 127.11.229.62:44945 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.229.2@KRBTEST.COM; defaulting to no policy
I20260923 17:28:46.453578 31414 ts_manager.cc:194] Registered new tserver with Master: 6af5566c43424d399cdb394fad67bca7 (127.11.229.1:37023)
I20260923 17:28:46.454941 31414 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.1@KRBTEST.COM'} at 127.11.229.1:48475
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab.
Entry for principal kudu/127.11.229.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab.
I20260923 17:28:46.472301 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.2:0
--local_ip_for_outbound_sockets=127.11.229.2
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:44945
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:37285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:46.584103 31610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:46.584457 31610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:46.584539 31610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:46.589771 31610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:46.589888 31610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:46.589988 31610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.2
I20260923 17:28:46.595709 31610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:37285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.2.keytab
--principal=kudu/127.11.229.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:44945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.2
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:46.596875 31610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:46.597959 31610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:46.607808 31616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:46.607800 31618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:46.607930 31610 server_base.cc:1034] running on GCE node
W20260923 17:28:46.607800 31615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:46.608469 31610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:46.609097 31610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:46.610282 31610 hybrid_clock.cc:648] HybridClock initialized: now 1790184526610263 us; error 29 us; skew 500 ppm
Sep 23 17:28:46 dist-test-slave-ptl9 krb5kdc[31369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184526, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:46.613040 31610 init.cc:377] Logged in from keytab as kudu/127.11.229.2@KRBTEST.COM (short username kudu)
I20260923 17:28:46.613986 31610 webserver.cc:460] Webserver started at http://127.11.229.2:40339/ using document root <none> and password file <none>
I20260923 17:28:46.614566 31610 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:46.614653 31610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:46.614879 31610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:46.616930 31610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-1/data/instance:
uuid: "228a991b4f8a44f9a2d1330f7adaffd2"
format_stamp: "Formatted at 2026-09-23 17:28:46 on dist-test-slave-ptl9"
server_key: "3cf60cbb326c9c0e4ee580207e0790b5"
server_key_iv: "0816e091dc7b88135e21193e02431458"
server_key_version: "encryptionkey@0"
I20260923 17:28:46.617527 31610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance:
uuid: "228a991b4f8a44f9a2d1330f7adaffd2"
format_stamp: "Formatted at 2026-09-23 17:28:46 on dist-test-slave-ptl9"
server_key: "3cf60cbb326c9c0e4ee580207e0790b5"
server_key_iv: "0816e091dc7b88135e21193e02431458"
server_key_version: "encryptionkey@0"
I20260923 17:28:46.621423 31610 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:28:46.623948 31625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:46.625254 31610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:28:46.625408 31610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-1/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-1/wal
uuid: "228a991b4f8a44f9a2d1330f7adaffd2"
format_stamp: "Formatted at 2026-09-23 17:28:46 on dist-test-slave-ptl9"
server_key: "3cf60cbb326c9c0e4ee580207e0790b5"
server_key_iv: "0816e091dc7b88135e21193e02431458"
server_key_version: "encryptionkey@0"
I20260923 17:28:46.625495 31610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:46.646803 31610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:46.650059 31610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:46.650221 31610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:46.650796 31610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:46.651602 31610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:46.651649 31610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:46.651691 31610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:46.651714 31610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:46.661311 31610 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.2:41817
I20260923 17:28:46.661331 31738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.2:41817 every 8 connection(s)
I20260923 17:28:46.662253 31610 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
I20260923 17:28:46.668798 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 31610
I20260923 17:28:46.668972 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance
I20260923 17:28:46.669318 12180 external_mini_cluster.cc:1428] Setting key 16dc26911846b62464cfaa0a542dba9f
Sep 23 17:28:46 dist-test-slave-ptl9 krb5kdc[31369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184526, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.2@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:28:46.678332 31453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:46.664146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:38905 (local address 127.11.229.62:44945)
0923 17:28:46.664281 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:46.664292 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:46.665221 (+ 929us) server_negotiation.cc:407] Connection header received
0923 17:28:46.666663 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:46.666666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:46.666727 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:46.666853 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:46.668127 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:46.668761 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:46.670321 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:46.670636 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:46.670741 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:46.674024 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:46.674032 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:46.674038 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:46.674077 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:46.675833 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:46.676427 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:46.676434 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:46.676439 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:46.676547 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:46.676914 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:46.676919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:46.676921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:46.677217 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:46.677390 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:46.677974 (+ 584us) server_negotiation.cc:299] Negotiation successful
0923 17:28:46.678122 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 17:28:46.679163 31741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:46.664455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:44945 (local address 127.11.229.2:38905)
0923 17:28:46.665040 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:46.665089 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:46.666428 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:46.667039 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:46.667047 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:46.667435 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:46.667971 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:46.667985 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:46.668986 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:46.668990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:46.670201 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:46.670212 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:46.670918 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:46.670927 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:46.671201 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:46.671895 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:46.671914 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:46.673796 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:46.675972 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:46.675978 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:46.675995 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:46.676283 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:46.676649 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:46.676651 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:46.676652 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:46.676792 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:46.677395 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:46.677401 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:46.677677 (+ 276us) client_negotiation.cc:769] Sending connection context
0923 17:28:46.677862 (+ 185us) client_negotiation.cc:240] Negotiation successful
0923 17:28:46.678160 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":124,"threads_started":1}
WARNING: no policy specified for kudu/127.11.229.3@KRBTEST.COM; defaulting to no policy
I20260923 17:28:46.680541 31739 heartbeater.cc:347] Connected to a master server at 127.11.229.62:44945
I20260923 17:28:46.680864 31739 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:46.681587 31739 heartbeater.cc:511] Master 127.11.229.62:44945 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.3@KRBTEST.COM" created.
I20260923 17:28:46.682983 31414 ts_manager.cc:194] Registered new tserver with Master: 228a991b4f8a44f9a2d1330f7adaffd2 (127.11.229.2:41817)
I20260923 17:28:46.683637 31414 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.2@KRBTEST.COM'} at 127.11.229.2:38905
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab.
Entry for principal kudu/127.11.229.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab.
I20260923 17:28:46.700650 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.3:0
--local_ip_for_outbound_sockets=127.11.229.3
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:44945
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:37285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:46.815415 31744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:46.815740 31744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:46.815817 31744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:46.819993 31744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:46.820091 31744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:46.820182 31744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.3
I20260923 17:28:46.825698 31744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:37285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.3.keytab
--principal=kudu/127.11.229.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:44945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.3
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:46.826881 31744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:46.827960 31744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:46.838091 31750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:46.838080 31749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:46.838080 31752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:46.838439 31744 server_base.cc:1034] running on GCE node
I20260923 17:28:46.838867 31744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:46.839563 31744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:46.840746 31744 hybrid_clock.cc:648] HybridClock initialized: now 1790184526840714 us; error 46 us; skew 500 ppm
Sep 23 17:28:46 dist-test-slave-ptl9 krb5kdc[31369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184526, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:46.843986 31744 init.cc:377] Logged in from keytab as kudu/127.11.229.3@KRBTEST.COM (short username kudu)
I20260923 17:28:46.845002 31744 webserver.cc:460] Webserver started at http://127.11.229.3:42113/ using document root <none> and password file <none>
I20260923 17:28:46.845666 31744 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:46.845757 31744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:46.845990 31744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:46.848161 31744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-2/data/instance:
uuid: "276accff714f4c3aa02583405593c73b"
format_stamp: "Formatted at 2026-09-23 17:28:46 on dist-test-slave-ptl9"
server_key: "45d5bce74a4aeb0a34d200f785e1d729"
server_key_iv: "820129a213d49a4d136eb5655724f6da"
server_key_version: "encryptionkey@0"
I20260923 17:28:46.848744 31744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance:
uuid: "276accff714f4c3aa02583405593c73b"
format_stamp: "Formatted at 2026-09-23 17:28:46 on dist-test-slave-ptl9"
server_key: "45d5bce74a4aeb0a34d200f785e1d729"
server_key_iv: "820129a213d49a4d136eb5655724f6da"
server_key_version: "encryptionkey@0"
I20260923 17:28:46.852612 31744 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 17:28:46.855459 31759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:46.856980 31744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:28:46.857113 31744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-2/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-2/wal
uuid: "276accff714f4c3aa02583405593c73b"
format_stamp: "Formatted at 2026-09-23 17:28:46 on dist-test-slave-ptl9"
server_key: "45d5bce74a4aeb0a34d200f785e1d729"
server_key_iv: "820129a213d49a4d136eb5655724f6da"
server_key_version: "encryptionkey@0"
I20260923 17:28:46.857244 31744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:46.876047 31744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:46.879599 31744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:46.879833 31744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:46.880498 31744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:46.881441 31744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:46.881511 31744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:46.881584 31744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:46.881630 31744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:46.890933 31744 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.3:42251
I20260923 17:28:46.890960 31872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.3:42251 every 8 connection(s)
I20260923 17:28:46.891868 31744 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
I20260923 17:28:46.897630 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 31744
I20260923 17:28:46.897776 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance
I20260923 17:28:46.898111 12180 external_mini_cluster.cc:1428] Setting key 6fff96cd6060c1201ef82addafcbfd03
Sep 23 17:28:46 dist-test-slave-ptl9 krb5kdc[31369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184526, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.3@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:28:46.906611 31453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:46.893941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:35867 (local address 127.11.229.62:44945)
0923 17:28:46.894115 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:46.894119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:46.894968 (+ 849us) server_negotiation.cc:407] Connection header received
0923 17:28:46.896359 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:46.896363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:46.896431 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:46.896554 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:46.897984 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:46.898571 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:46.900127 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:46.900462 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:46.900579 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:46.903233 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:46.903236 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:46.903238 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:46.903264 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:46.904476 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:46.904973 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:46.904979 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:46.904981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:46.905066 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:46.905488 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:46.905491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:46.905493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:46.905688 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:46.905815 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:46.906321 (+ 506us) server_negotiation.cc:299] Negotiation successful
0923 17:28:46.906445 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 17:28:46.907450 31875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:46.894305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:44945 (local address 127.11.229.3:35867)
0923 17:28:46.894795 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:46.894842 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:46.896121 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:46.896729 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:46.896736 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:46.897109 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:46.897812 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:46.897835 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:46.898681 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:46.898684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:46.899997 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:46.900007 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:46.900621 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:46.900625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:46.900809 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:46.901487 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:46.901504 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:46.903076 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:46.904582 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:46.904587 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:46.904599 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:46.904859 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:46.905171 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:46.905174 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:46.905175 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:46.905385 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:46.905815 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:46.905823 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:46.906076 (+ 253us) client_negotiation.cc:769] Sending connection context
0923 17:28:46.906271 (+ 195us) client_negotiation.cc:240] Negotiation successful
0923 17:28:46.906603 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":131,"threads_started":1}
I20260923 17:28:46.908603 31873 heartbeater.cc:347] Connected to a master server at 127.11.229.62:44945
I20260923 17:28:46.908864 31873 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:46.909508 31873 heartbeater.cc:511] Master 127.11.229.62:44945 requested a full tablet report, sending...
I20260923 17:28:46.910630 31414 ts_manager.cc:194] Registered new tserver with Master: 276accff714f4c3aa02583405593c73b (127.11.229.3:42251)
I20260923 17:28:46.911445 31414 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.229.3@KRBTEST.COM'} at 127.11.229.3:35867
I20260923 17:28:46.912246 12180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:28:46.922118 31453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:46.914573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58188 (local address 127.11.229.62:44945)
0923 17:28:46.914750 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:46.914754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:46.914848 (+ 94us) server_negotiation.cc:407] Connection header received
0923 17:28:46.914954 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:46.914957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:46.914996 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:46.915063 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:46.915724 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:46.916332 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:46.917250 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:46.917559 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:46.917649 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:46.919134 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:46.919138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:46.919140 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:46.919165 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:46.920443 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:46.920875 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:46.920879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:46.920881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:46.920965 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:46.921201 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:46.921204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:46.921205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:46.921474 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:46.921581 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:46.921780 (+ 199us) server_negotiation.cc:299] Negotiation successful
0923 17:28:46.921886 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260923 17:28:46.925400 31414 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:28:46.929382 31414 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:28:46.944521 31885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:46.938430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44062 (local address 127.11.229.2:41817)
0923 17:28:46.939365 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:46.939380 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:46.939405 (+ 25us) server_negotiation.cc:407] Connection header received
0923 17:28:46.939487 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:46.939494 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:46.939722 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:46.939927 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:46.940297 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:46.941612 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:46.942998 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:46.943861 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:46.943976 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:46.944127 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:46.944304 (+ 177us) server_negotiation.cc:299] Negotiation successful
0923 17:28:46.944391 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":826,"thread_start_us":122,"threads_started":1}
I20260923 17:28:46.944582 31884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:46.938234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:41817 (local address 127.0.0.1:44062)
0923 17:28:46.938637 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:46.938696 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:46.938827 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:46.939928 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:46.939934 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:46.939955 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:46.940136 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:46.940145 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:46.941757 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:46.941760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:46.942869 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:46.942880 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:46.943989 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:46.943993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:46.944102 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:46.944155 (+ 53us) client_negotiation.cc:769] Sending connection context
0923 17:28:46.944278 (+ 123us) client_negotiation.cc:240] Negotiation successful
0923 17:28:46.944417 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":47,"thread_start_us":122,"threads_started":1}
I20260923 17:28:46.945041 31886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:46.938594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:37023 (local address 127.0.0.1:41974)
0923 17:28:46.939630 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:46.939643 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:46.939726 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:46.940417 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:46.940419 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:46.940439 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:46.940576 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:46.940583 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:46.941665 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:46.941668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:46.943587 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:46.943594 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:46.944606 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:46.944613 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:46.944731 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:46.944760 (+ 29us) client_negotiation.cc:769] Sending connection context
0923 17:28:46.944843 (+ 83us) client_negotiation.cc:240] Negotiation successful
0923 17:28:46.944909 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":974,"mutex_wait_us":23,"spinlock_wait_cycles":896,"thread_start_us":110,"threads_started":1}
I20260923 17:28:46.945319 31889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:46.939227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59200 (local address 127.11.229.3:42251)
0923 17:28:46.939637 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:46.939653 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:46.939793 (+ 140us) server_negotiation.cc:407] Connection header received
0923 17:28:46.939957 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:46.939962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:46.940141 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:46.940279 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:46.940741 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:46.941847 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:46.943072 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:46.943862 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:46.943976 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:46.944127 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:46.945068 (+ 941us) server_negotiation.cc:299] Negotiation successful
0923 17:28:46.945165 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":127,"threads_started":1}
I20260923 17:28:46.945245 31887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:46.938994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:42251 (local address 127.0.0.1:59200)
0923 17:28:46.939691 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:46.939709 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:46.939820 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:46.940450 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:46.940453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:46.940483 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:46.940618 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:46.940624 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:46.941991 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:46.941994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:46.942909 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:46.942919 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:46.944808 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:46.944812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:46.944918 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:46.944944 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 17:28:46.945032 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 17:28:46.945105 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":31,"spinlock_wait_cycles":768,"thread_start_us":90,"threads_started":1}
I20260923 17:28:46.945537 31888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:46.939064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41974 (local address 127.11.229.1:37023)
0923 17:28:46.939998 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:46.940013 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:46.940034 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:28:46.940127 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:46.940132 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:46.940289 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:46.940438 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:46.940702 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:46.941501 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:46.943700 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:46.944477 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:46.944582 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:46.944734 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:46.945288 (+ 554us) server_negotiation.cc:299] Negotiation successful
0923 17:28:46.945396 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":128,"threads_started":1}
I20260923 17:28:46.948228 31807 tablet_service.cc:1467] Processing CreateTablet for tablet 27dd0524d33a488687ee7db069be7012 (DEFAULT_TABLE table=test-table [id=97fa7a9ce5d74e3aa37fc7af4c21274a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:46.948330 31673 tablet_service.cc:1467] Processing CreateTablet for tablet 27dd0524d33a488687ee7db069be7012 (DEFAULT_TABLE table=test-table [id=97fa7a9ce5d74e3aa37fc7af4c21274a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:46.948338 31539 tablet_service.cc:1467] Processing CreateTablet for tablet 27dd0524d33a488687ee7db069be7012 (DEFAULT_TABLE table=test-table [id=97fa7a9ce5d74e3aa37fc7af4c21274a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:46.949395 31673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27dd0524d33a488687ee7db069be7012. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:46.949395 31539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27dd0524d33a488687ee7db069be7012. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:46.949404 31807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27dd0524d33a488687ee7db069be7012. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:46.956270 31890 tablet_bootstrap.cc:492] T 27dd0524d33a488687ee7db069be7012 P 276accff714f4c3aa02583405593c73b: Bootstrap starting.
I20260923 17:28:46.957628 31892 tablet_bootstrap.cc:492] T 27dd0524d33a488687ee7db069be7012 P 228a991b4f8a44f9a2d1330f7adaffd2: Bootstrap starting.
I20260923 17:28:46.958633 31890 tablet_bootstrap.cc:654] T 27dd0524d33a488687ee7db069be7012 P 276accff714f4c3aa02583405593c73b: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:46.959704 31892 tablet_bootstrap.cc:654] T 27dd0524d33a488687ee7db069be7012 P 228a991b4f8a44f9a2d1330f7adaffd2: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:46.959786 31890 log.cc:824] T 27dd0524d33a488687ee7db069be7012 P 276accff714f4c3aa02583405593c73b: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:46.960554 31892 log.cc:824] T 27dd0524d33a488687ee7db069be7012 P 228a991b4f8a44f9a2d1330f7adaffd2: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:46.960750 31891 tablet_bootstrap.cc:492] T 27dd0524d33a488687ee7db069be7012 P 6af5566c43424d399cdb394fad67bca7: Bootstrap starting.
I20260923 17:28:46.962328 31890 tablet_bootstrap.cc:492] T 27dd0524d33a488687ee7db069be7012 P 276accff714f4c3aa02583405593c73b: No bootstrap required, opened a new log
I20260923 17:28:46.962539 31890 ts_tablet_manager.cc:1397] T 27dd0524d33a488687ee7db069be7012 P 276accff714f4c3aa02583405593c73b: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260923 17:28:46.962925 31891 tablet_bootstrap.cc:654] T 27dd0524d33a488687ee7db069be7012 P 6af5566c43424d399cdb394fad67bca7: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:46.963006 31892 tablet_bootstrap.cc:492] T 27dd0524d33a488687ee7db069be7012 P 228a991b4f8a44f9a2d1330f7adaffd2: No bootstrap required, opened a new log
I20260923 17:28:46.963186 31892 ts_tablet_manager.cc:1397] T 27dd0524d33a488687ee7db069be7012 P 228a991b4f8a44f9a2d1330f7adaffd2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 17:28:46.963647 31891 log.cc:824] T 27dd0524d33a488687ee7db069be7012 P 6af5566c43424d399cdb394fad67bca7: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:46.965242 31891 tablet_bootstrap.cc:492] T 27dd0524d33a488687ee7db069be7012 P 6af5566c43424d399cdb394fad67bca7: No bootstrap required, opened a new log
I20260923 17:28:46.965502 31891 ts_tablet_manager.cc:1397] T 27dd0524d33a488687ee7db069be7012 P 6af5566c43424d399cdb394fad67bca7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260923 17:28:46.966912 31892 raft_consensus.cc:348] T 27dd0524d33a488687ee7db069be7012 P 228a991b4f8a44f9a2d1330f7adaffd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "228a991b4f8a44f9a2d1330f7adaffd2" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 41817 } } peers { permanent_uuid: "276accff714f4c3aa02583405593c73b" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 42251 } } peers { permanent_uuid: "6af5566c43424d399cdb394fad67bca7" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 37023 } }
I20260923 17:28:46.967171 31892 raft_consensus.cc:374] T 27dd0524d33a488687ee7db069be7012 P 228a991b4f8a44f9a2d1330f7adaffd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:46.967231 31892 raft_consensus.cc:729] T 27dd0524d33a488687ee7db069be7012 P 228a991b4f8a44f9a2d1330f7adaffd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 228a991b4f8a44f9a2d1330f7adaffd2, State: Initialized, Role: FOLLOWER
I20260923 17:28:46.966624 31890 raft_consensus.cc:348] T 27dd0524d33a488687ee7db069be7012 P 276accff714f4c3aa02583405593c73b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "228a991b4f8a44f9a2d1330f7adaffd2" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 41817 } } peers { permanent_uuid: "276accff714f4c3aa02583405593c73b" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 42251 } } peers { permanent_uuid: "6af5566c43424d399cdb394fad67bca7" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 37023 } }
I20260923 17:28:46.967747 31892 consensus_queue.cc:260] T 27dd0524d33a488687ee7db069be7012 P 228a991b4f8a44f9a2d1330f7adaffd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "228a991b4f8a44f9a2d1330f7adaffd2" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 41817 } } peers { permanent_uuid: "276accff714f4c3aa02583405593c73b" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 42251 } } peers { permanent_uuid: "6af5566c43424d399cdb394fad67bca7" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 37023 } }
I20260923 17:28:46.967867 31890 raft_consensus.cc:374] T 27dd0524d33a488687ee7db069be7012 P 276accff714f4c3aa02583405593c73b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:46.967919 31890 raft_consensus.cc:729] T 27dd0524d33a488687ee7db069be7012 P 276accff714f4c3aa02583405593c73b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 276accff714f4c3aa02583405593c73b, State: Initialized, Role: FOLLOWER
I20260923 17:28:46.968394 31890 consensus_queue.cc:260] T 27dd0524d33a488687ee7db069be7012 P 276accff714f4c3aa02583405593c73b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "228a991b4f8a44f9a2d1330f7adaffd2" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 41817 } } peers { permanent_uuid: "276accff714f4c3aa02583405593c73b" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 42251 } } peers { permanent_uuid: "6af5566c43424d399cdb394fad67bca7" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 37023 } }
I20260923 17:28:46.968467 31739 heartbeater.cc:503] Master 127.11.229.62:44945 was elected leader, sending a full tablet report...
I20260923 17:28:46.968850 31892 ts_tablet_manager.cc:1428] T 27dd0524d33a488687ee7db069be7012 P 228a991b4f8a44f9a2d1330f7adaffd2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 17:28:46.969074 31873 heartbeater.cc:503] Master 127.11.229.62:44945 was elected leader, sending a full tablet report...
I20260923 17:28:46.969360 31890 ts_tablet_manager.cc:1428] T 27dd0524d33a488687ee7db069be7012 P 276accff714f4c3aa02583405593c73b: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260923 17:28:46.970494 31891 raft_consensus.cc:348] T 27dd0524d33a488687ee7db069be7012 P 6af5566c43424d399cdb394fad67bca7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "228a991b4f8a44f9a2d1330f7adaffd2" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 41817 } } peers { permanent_uuid: "276accff714f4c3aa02583405593c73b" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 42251 } } peers { permanent_uuid: "6af5566c43424d399cdb394fad67bca7" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 37023 } }
I20260923 17:28:46.970705 31891 raft_consensus.cc:374] T 27dd0524d33a488687ee7db069be7012 P 6af5566c43424d399cdb394fad67bca7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:46.970769 31891 raft_consensus.cc:729] T 27dd0524d33a488687ee7db069be7012 P 6af5566c43424d399cdb394fad67bca7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af5566c43424d399cdb394fad67bca7, State: Initialized, Role: FOLLOWER
I20260923 17:28:46.971119 31891 consensus_queue.cc:260] T 27dd0524d33a488687ee7db069be7012 P 6af5566c43424d399cdb394fad67bca7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "228a991b4f8a44f9a2d1330f7adaffd2" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 41817 } } peers { permanent_uuid: "276accff714f4c3aa02583405593c73b" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 42251 } } peers { permanent_uuid: "6af5566c43424d399cdb394fad67bca7" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 37023 } }
I20260923 17:28:46.971750 31605 heartbeater.cc:503] Master 127.11.229.62:44945 was elected leader, sending a full tablet report...
I20260923 17:28:46.971920 31891 ts_tablet_manager.cc:1428] T 27dd0524d33a488687ee7db069be7012 P 6af5566c43424d399cdb394fad67bca7: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
W20260923 17:28:47.142858 31874 tablet.cc:2369] T 27dd0524d33a488687ee7db069be7012 P 276accff714f4c3aa02583405593c73b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:28:47.163463 31740 tablet.cc:2369] T 27dd0524d33a488687ee7db069be7012 P 228a991b4f8a44f9a2d1330f7adaffd2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:28:47.185547 31606 tablet.cc:2369] T 27dd0524d33a488687ee7db069be7012 P 6af5566c43424d399cdb394fad67bca7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:28:47.189628 31896 raft_consensus.cc:482] T 27dd0524d33a488687ee7db069be7012 P 228a991b4f8a44f9a2d1330f7adaffd2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:28:47.189838 31896 raft_consensus.cc:504] T 27dd0524d33a488687ee7db069be7012 P 228a991b4f8a44f9a2d1330f7adaffd2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "228a991b4f8a44f9a2d1330f7adaffd2" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 41817 } } peers { permanent_uuid: "276accff714f4c3aa02583405593c73b" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 42251 } } peers { permanent_uuid: "6af5566c43424d399cdb394fad67bca7" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 37023 } }
I20260923 17:28:47.190999 31896 leader_election.cc:288] T 27dd0524d33a488687ee7db069be7012 P 228a991b4f8a44f9a2d1330f7adaffd2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 276accff714f4c3aa02583405593c73b (127.11.229.3:42251), 6af5566c43424d399cdb394fad67bca7 (127.11.229.1:37023)
I20260923 17:28:47.195183 31888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:47.191391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:39053 (local address 127.11.229.1:37023)
0923 17:28:47.191552 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:47.191557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:47.191731 (+ 174us) server_negotiation.cc:407] Connection header received
0923 17:28:47.191942 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:47.191944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:47.191989 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:47.192077 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:47.192362 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:47.192882 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:47.194111 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:47.194542 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:47.194670 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:47.194727 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:47.194970 (+ 243us) server_negotiation.cc:299] Negotiation successful
0923 17:28:47.195044 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 17:28:47.195183 31889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:47.191342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:51771 (local address 127.11.229.3:42251)
0923 17:28:47.191487 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:47.191493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:47.191745 (+ 252us) server_negotiation.cc:407] Connection header received
0923 17:28:47.191940 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:47.191943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:47.191992 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:47.192071 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:47.192381 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:47.192879 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:47.194113 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:47.194536 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:47.194666 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:47.194726 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:47.194970 (+ 244us) server_negotiation.cc:299] Negotiation successful
0923 17:28:47.195041 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 17:28:47.195211 31900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:47.191288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:37023 (local address 127.11.229.2:39053)
0923 17:28:47.191626 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:47.191638 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:47.191817 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:47.192090 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:47.192093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:47.192117 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:47.192252 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:47.192257 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:47.192997 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:47.193002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:47.193987 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:47.193996 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:47.194666 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:47.194670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:47.194807 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:47.194858 (+ 51us) client_negotiation.cc:769] Sending connection context
0923 17:28:47.194943 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 17:28:47.195037 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":82,"spinlock_wait_cycles":35200,"thread_start_us":115,"threads_started":1}
I20260923 17:28:47.195226 31899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:47.191216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:42251 (local address 127.11.229.2:51771)
0923 17:28:47.191626 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:47.191644 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:47.191819 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:47.192096 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:47.192099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:47.192121 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:47.192265 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:47.192270 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:47.192986 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:47.192991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:47.193985 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:47.193995 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:47.194666 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:47.194669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:47.194815 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:47.194865 (+ 50us) client_negotiation.cc:769] Sending connection context
0923 17:28:47.194952 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 17:28:47.195041 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":83,"spinlock_wait_cycles":8704,"thread_start_us":115,"threads_started":1}
I20260923 17:28:47.196020 31559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27dd0524d33a488687ee7db069be7012" candidate_uuid: "228a991b4f8a44f9a2d1330f7adaffd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af5566c43424d399cdb394fad67bca7" is_pre_election: true
I20260923 17:28:47.196105 31827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27dd0524d33a488687ee7db069be7012" candidate_uuid: "228a991b4f8a44f9a2d1330f7adaffd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "276accff714f4c3aa02583405593c73b" is_pre_election: true
I20260923 17:28:47.196394 31559 raft_consensus.cc:2454] T 27dd0524d33a488687ee7db069be7012 P 6af5566c43424d399cdb394fad67bca7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 228a991b4f8a44f9a2d1330f7adaffd2 in term 0.
I20260923 17:28:47.196419 31827 raft_consensus.cc:2454] T 27dd0524d33a488687ee7db069be7012 P 276accff714f4c3aa02583405593c73b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 228a991b4f8a44f9a2d1330f7adaffd2 in term 0.
I20260923 17:28:47.197028 31627 leader_election.cc:302] T 27dd0524d33a488687ee7db069be7012 P 228a991b4f8a44f9a2d1330f7adaffd2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 228a991b4f8a44f9a2d1330f7adaffd2, 276accff714f4c3aa02583405593c73b, 6af5566c43424d399cdb394fad67bca7; no voters:
I20260923 17:28:47.197468 31896 raft_consensus.cc:2781] T 27dd0524d33a488687ee7db069be7012 P 228a991b4f8a44f9a2d1330f7adaffd2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:28:47.197553 31896 raft_consensus.cc:482] T 27dd0524d33a488687ee7db069be7012 P 228a991b4f8a44f9a2d1330f7adaffd2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:28:47.197587 31896 raft_consensus.cc:3037] T 27dd0524d33a488687ee7db069be7012 P 228a991b4f8a44f9a2d1330f7adaffd2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:47.198696 31896 raft_consensus.cc:504] T 27dd0524d33a488687ee7db069be7012 P 228a991b4f8a44f9a2d1330f7adaffd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "228a991b4f8a44f9a2d1330f7adaffd2" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 41817 } } peers { permanent_uuid: "276accff714f4c3aa02583405593c73b" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 42251 } } peers { permanent_uuid: "6af5566c43424d399cdb394fad67bca7" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 37023 } }
I20260923 17:28:47.199159 31896 leader_election.cc:288] T 27dd0524d33a488687ee7db069be7012 P 228a991b4f8a44f9a2d1330f7adaffd2 [CANDIDATE]: Term 1 election: Requested vote from peers 276accff714f4c3aa02583405593c73b (127.11.229.3:42251), 6af5566c43424d399cdb394fad67bca7 (127.11.229.1:37023)
I20260923 17:28:47.199576 31827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27dd0524d33a488687ee7db069be7012" candidate_uuid: "228a991b4f8a44f9a2d1330f7adaffd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "276accff714f4c3aa02583405593c73b"
I20260923 17:28:47.199599 31559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27dd0524d33a488687ee7db069be7012" candidate_uuid: "228a991b4f8a44f9a2d1330f7adaffd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af5566c43424d399cdb394fad67bca7"
I20260923 17:28:47.199697 31827 raft_consensus.cc:3037] T 27dd0524d33a488687ee7db069be7012 P 276accff714f4c3aa02583405593c73b [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:47.199756 31559 raft_consensus.cc:3037] T 27dd0524d33a488687ee7db069be7012 P 6af5566c43424d399cdb394fad67bca7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:47.200876 31827 raft_consensus.cc:2454] T 27dd0524d33a488687ee7db069be7012 P 276accff714f4c3aa02583405593c73b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 228a991b4f8a44f9a2d1330f7adaffd2 in term 1.
I20260923 17:28:47.200888 31559 raft_consensus.cc:2454] T 27dd0524d33a488687ee7db069be7012 P 6af5566c43424d399cdb394fad67bca7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 228a991b4f8a44f9a2d1330f7adaffd2 in term 1.
I20260923 17:28:47.201221 31627 leader_election.cc:302] T 27dd0524d33a488687ee7db069be7012 P 228a991b4f8a44f9a2d1330f7adaffd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 228a991b4f8a44f9a2d1330f7adaffd2, 276accff714f4c3aa02583405593c73b; no voters:
I20260923 17:28:47.201462 31896 raft_consensus.cc:2781] T 27dd0524d33a488687ee7db069be7012 P 228a991b4f8a44f9a2d1330f7adaffd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:47.201692 31896 raft_consensus.cc:686] T 27dd0524d33a488687ee7db069be7012 P 228a991b4f8a44f9a2d1330f7adaffd2 [term 1 LEADER]: Becoming Leader. State: Replica: 228a991b4f8a44f9a2d1330f7adaffd2, State: Running, Role: LEADER
I20260923 17:28:47.202091 31896 consensus_queue.cc:237] T 27dd0524d33a488687ee7db069be7012 P 228a991b4f8a44f9a2d1330f7adaffd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "228a991b4f8a44f9a2d1330f7adaffd2" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 41817 } } peers { permanent_uuid: "276accff714f4c3aa02583405593c73b" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 42251 } } peers { permanent_uuid: "6af5566c43424d399cdb394fad67bca7" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 37023 } }
I20260923 17:28:47.205582 31414 catalog_manager.cc:5495] T 27dd0524d33a488687ee7db069be7012 P 228a991b4f8a44f9a2d1330f7adaffd2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 228a991b4f8a44f9a2d1330f7adaffd2 (127.11.229.2). New cstate: current_term: 1 leader_uuid: "228a991b4f8a44f9a2d1330f7adaffd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "228a991b4f8a44f9a2d1330f7adaffd2" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "276accff714f4c3aa02583405593c73b" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 42251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6af5566c43424d399cdb394fad67bca7" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 37023 } health_report { overall_health: UNKNOWN } } }
I20260923 17:28:47.238126 31885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:47.234208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44074 (local address 127.11.229.2:41817)
0923 17:28:47.234383 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:47.234387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:47.234401 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:28:47.234484 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:47.234486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:47.234544 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:47.234685 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:28:47.234988 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:47.235510 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:47.236424 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:47.236650 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:47.236743 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:47.237153 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:28:47.237682 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:28:47.237785 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:47.237945 (+ 160us) server_negotiation.cc:299] Negotiation successful
0923 17:28:47.238005 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260923 17:28:47.244488 31559 raft_consensus.cc:1261] T 27dd0524d33a488687ee7db069be7012 P 6af5566c43424d399cdb394fad67bca7 [term 1 FOLLOWER]: Refusing update from remote peer 228a991b4f8a44f9a2d1330f7adaffd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:28:47.244676 31827 raft_consensus.cc:1261] T 27dd0524d33a488687ee7db069be7012 P 276accff714f4c3aa02583405593c73b [term 1 FOLLOWER]: Refusing update from remote peer 228a991b4f8a44f9a2d1330f7adaffd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:28:47.245404 31896 consensus_queue.cc:1035] T 27dd0524d33a488687ee7db069be7012 P 228a991b4f8a44f9a2d1330f7adaffd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "276accff714f4c3aa02583405593c73b" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 42251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:28:47.245666 31901 consensus_queue.cc:1035] T 27dd0524d33a488687ee7db069be7012 P 228a991b4f8a44f9a2d1330f7adaffd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6af5566c43424d399cdb394fad67bca7" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 37023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:28:47.256062 31906 mvcc.cc:205] Tried to move back new op lower bound from 7332595823585648640 to 7332595823422582784. Current Snapshot: MvccSnapshot[applied={T|T < 7332595823585648640}]
I20260923 17:28:47.257310 31905 mvcc.cc:205] Tried to move back new op lower bound from 7332595823585648640 to 7332595823422582784. Current Snapshot: MvccSnapshot[applied={T|T < 7332595823585648640}]
I20260923 17:28:47.265499 31413 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:28:47.265722 31413 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:28:47.268713 31413 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 423705948a254af0acefc05917047ae3: Sending DeleteTablet for 3 replicas of tablet 27dd0524d33a488687ee7db069be7012
I20260923 17:28:47.269795 31539 tablet_service.cc:1514] Processing DeleteTablet for tablet 27dd0524d33a488687ee7db069be7012 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:47 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:41974
I20260923 17:28:47.269789 31673 tablet_service.cc:1514] Processing DeleteTablet for tablet 27dd0524d33a488687ee7db069be7012 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:47 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:44062
I20260923 17:28:47.270427 31915 tablet_replica.cc:330] T 27dd0524d33a488687ee7db069be7012 P 6af5566c43424d399cdb394fad67bca7: stopping tablet replica
I20260923 17:28:47.270434 31807 tablet_service.cc:1514] Processing DeleteTablet for tablet 27dd0524d33a488687ee7db069be7012 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:47 UTC) from {username='kudu', principal='kudu/127.11.229.62@KRBTEST.COM'} at 127.0.0.1:59200
I20260923 17:28:47.270779 31916 tablet_replica.cc:330] T 27dd0524d33a488687ee7db069be7012 P 228a991b4f8a44f9a2d1330f7adaffd2: stopping tablet replica
I20260923 17:28:47.270957 31917 tablet_replica.cc:330] T 27dd0524d33a488687ee7db069be7012 P 276accff714f4c3aa02583405593c73b: stopping tablet replica
I20260923 17:28:47.271245 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 31476
I20260923 17:28:47.271317 31917 raft_consensus.cc:2229] T 27dd0524d33a488687ee7db069be7012 P 276accff714f4c3aa02583405593c73b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:28:47.271552 31916 raft_consensus.cc:2229] T 27dd0524d33a488687ee7db069be7012 P 228a991b4f8a44f9a2d1330f7adaffd2 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:28:47.271688 31917 raft_consensus.cc:2258] T 27dd0524d33a488687ee7db069be7012 P 276accff714f4c3aa02583405593c73b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:47.273801 31917 ts_tablet_manager.cc:1905] T 27dd0524d33a488687ee7db069be7012 P 276accff714f4c3aa02583405593c73b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:47.274057 31916 raft_consensus.cc:2258] T 27dd0524d33a488687ee7db069be7012 P 228a991b4f8a44f9a2d1330f7adaffd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:47.277452 31917 ts_tablet_manager.cc:1918] T 27dd0524d33a488687ee7db069be7012 P 276accff714f4c3aa02583405593c73b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:28:47.277693 31917 log.cc:1196] T 27dd0524d33a488687ee7db069be7012 P 276accff714f4c3aa02583405593c73b: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-2/wal/wals/27dd0524d33a488687ee7db069be7012
I20260923 17:28:47.277796 31916 ts_tablet_manager.cc:1905] T 27dd0524d33a488687ee7db069be7012 P 228a991b4f8a44f9a2d1330f7adaffd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:47.278210 31917 ts_tablet_manager.cc:1939] T 27dd0524d33a488687ee7db069be7012 P 276accff714f4c3aa02583405593c73b: Deleting consensus metadata
I20260923 17:28:47.279819 31399 catalog_manager.cc:4841] TS 276accff714f4c3aa02583405593c73b (127.11.229.3:42251): tablet 27dd0524d33a488687ee7db069be7012 (table test-table [id=97fa7a9ce5d74e3aa37fc7af4c21274a]) successfully deleted
W20260923 17:28:47.280606 31401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.229.1:37023 (error 108)
I20260923 17:28:47.281329 31916 ts_tablet_manager.cc:1918] T 27dd0524d33a488687ee7db069be7012 P 228a991b4f8a44f9a2d1330f7adaffd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:28:47.281448 31916 log.cc:1196] T 27dd0524d33a488687ee7db069be7012 P 228a991b4f8a44f9a2d1330f7adaffd2: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184271597484-12180-0/minicluster-data/ts-1/wal/wals/27dd0524d33a488687ee7db069be7012
I20260923 17:28:47.281781 31916 ts_tablet_manager.cc:1939] T 27dd0524d33a488687ee7db069be7012 P 228a991b4f8a44f9a2d1330f7adaffd2: Deleting consensus metadata
I20260923 17:28:47.281956 31886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:47.281483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:37023 (local address 127.0.0.1:41990)
0923 17:28:47.281694 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:47.281769 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.1:37023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260923 17:28:47.282141 31401 catalog_manager.cc:4584] TS 6af5566c43424d399cdb394fad67bca7 (127.11.229.1:37023): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 27dd0524d33a488687ee7db069be7012: Network error: Client connection negotiation failed: client connection to 127.11.229.1:37023: connect: Connection refused (error 111)
I20260923 17:28:47.282621 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 31610
I20260923 17:28:47.289487 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 31744
I20260923 17:28:47.290100 31886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:47.289728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:41817 (local address 127.0.0.1:44080)
0923 17:28:47.289934 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:47.289991 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.2:41817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260923 17:28:47.295953 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 31382
2026-09-23T17:28:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2516 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-23T17:28:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:28:47Z Disabled control of system clock
I20260923 17:28:47.327668 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:39211
--webserver_interface=127.11.229.62
--webserver_port=0
--builtin_ntp_servers=127.11.229.20:41455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:39211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 17:28:47.443017 31922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:47.443300 31922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:47.443346 31922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:47.447500 31922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:28:47.447573 31922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:47.447589 31922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:47.447607 31922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:28:47.447623 31922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:28:47.452570 31922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:41455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:39211
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:39211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:47.453894 31922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:47.455331 31922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:47.464254 31930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:47.464285 31928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:47.464293 31927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:47.464334 31922 server_base.cc:1034] running on GCE node
I20260923 17:28:47.465235 31922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:47.466457 31922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:47.467641 31922 hybrid_clock.cc:648] HybridClock initialized: now 1790184527467620 us; error 33 us; skew 500 ppm
I20260923 17:28:47.469630 31922 webserver.cc:460] Webserver started at http://127.11.229.62:43391/ using document root <none> and password file <none>
I20260923 17:28:47.470203 31922 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:47.470263 31922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:47.470427 31922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:47.472450 31922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "4cf83234843c40a693b6ce0211ad5e8f"
format_stamp: "Formatted at 2026-09-23 17:28:47 on dist-test-slave-ptl9"
server_key: "47c85b5526ad93b9d986d21b5e07d971"
server_key_iv: "0cf0233c644e6d9d20a540676fc3b4fc"
server_key_version: "encryptionkey@0"
I20260923 17:28:47.472976 31922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "4cf83234843c40a693b6ce0211ad5e8f"
format_stamp: "Formatted at 2026-09-23 17:28:47 on dist-test-slave-ptl9"
server_key: "47c85b5526ad93b9d986d21b5e07d971"
server_key_iv: "0cf0233c644e6d9d20a540676fc3b4fc"
server_key_version: "encryptionkey@0"
I20260923 17:28:47.476718 31922 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260923 17:28:47.479159 31936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:47.480414 31922 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:28:47.480517 31922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "4cf83234843c40a693b6ce0211ad5e8f"
format_stamp: "Formatted at 2026-09-23 17:28:47 on dist-test-slave-ptl9"
server_key: "47c85b5526ad93b9d986d21b5e07d971"
server_key_iv: "0cf0233c644e6d9d20a540676fc3b4fc"
server_key_version: "encryptionkey@0"
I20260923 17:28:47.480605 31922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:47.519380 31922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:47.520093 31922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:47.520262 31922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:47.527323 31922 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:39211
I20260923 17:28:47.527346 31988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:39211 every 8 connection(s)
I20260923 17:28:47.528306 31922 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:28:47.531541 31989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:47.534272 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 31922
I20260923 17:28:47.534417 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:28:47.534732 12180 external_mini_cluster.cc:1428] Setting key 6de2717f0c87b993f3acf831742df35b
I20260923 17:28:47.538232 31989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cf83234843c40a693b6ce0211ad5e8f: Bootstrap starting.
I20260923 17:28:47.540449 31989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cf83234843c40a693b6ce0211ad5e8f: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:47.541188 31989 log.cc:824] T 00000000000000000000000000000000 P 4cf83234843c40a693b6ce0211ad5e8f: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:47.543383 31989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cf83234843c40a693b6ce0211ad5e8f: No bootstrap required, opened a new log
I20260923 17:28:47.544294 31992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:47.536307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55238 (local address 127.11.229.62:39211)
0923 17:28:47.537006 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:47.537021 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:47.537058 (+ 37us) server_negotiation.cc:407] Connection header received
0923 17:28:47.537763 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:47.537785 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:47.538185 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:47.538589 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:47.539031 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:47.539982 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:47.540935 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:47.541475 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:47.542111 (+ 636us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:47.542538 (+ 427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:47.542542 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:47.542555 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:47.542935 (+ 380us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:47.542992 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:47.543012 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:47.543088 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:47.543255 (+ 167us) server_negotiation.cc:299] Negotiation successful
0923 17:28:47.543592 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":133,"threads_started":1}
I20260923 17:28:47.547475 31989 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cf83234843c40a693b6ce0211ad5e8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cf83234843c40a693b6ce0211ad5e8f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 39211 } }
I20260923 17:28:47.547746 31989 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4cf83234843c40a693b6ce0211ad5e8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:47.547820 31989 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cf83234843c40a693b6ce0211ad5e8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cf83234843c40a693b6ce0211ad5e8f, State: Initialized, Role: FOLLOWER
I20260923 17:28:47.548316 31989 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cf83234843c40a693b6ce0211ad5e8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cf83234843c40a693b6ce0211ad5e8f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 39211 } }
I20260923 17:28:47.548482 31989 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4cf83234843c40a693b6ce0211ad5e8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:28:47.548558 31989 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cf83234843c40a693b6ce0211ad5e8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:28:47.548666 31989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cf83234843c40a693b6ce0211ad5e8f [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:47.549863 31989 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cf83234843c40a693b6ce0211ad5e8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cf83234843c40a693b6ce0211ad5e8f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 39211 } }
I20260923 17:28:47.550225 31989 leader_election.cc:302] T 00000000000000000000000000000000 P 4cf83234843c40a693b6ce0211ad5e8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cf83234843c40a693b6ce0211ad5e8f; no voters:
I20260923 17:28:47.550575 31989 leader_election.cc:288] T 00000000000000000000000000000000 P 4cf83234843c40a693b6ce0211ad5e8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:28:47.550710 31994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cf83234843c40a693b6ce0211ad5e8f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:47.550974 31994 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4cf83234843c40a693b6ce0211ad5e8f [term 1 LEADER]: Becoming Leader. State: Replica: 4cf83234843c40a693b6ce0211ad5e8f, State: Running, Role: LEADER
I20260923 17:28:47.551319 31994 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cf83234843c40a693b6ce0211ad5e8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cf83234843c40a693b6ce0211ad5e8f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 39211 } }
I20260923 17:28:47.551851 31989 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cf83234843c40a693b6ce0211ad5e8f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:28:47.553110 31996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cf83234843c40a693b6ce0211ad5e8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cf83234843c40a693b6ce0211ad5e8f. Latest consensus state: current_term: 1 leader_uuid: "4cf83234843c40a693b6ce0211ad5e8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cf83234843c40a693b6ce0211ad5e8f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 39211 } } }
I20260923 17:28:47.553110 31995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cf83234843c40a693b6ce0211ad5e8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cf83234843c40a693b6ce0211ad5e8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cf83234843c40a693b6ce0211ad5e8f" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 39211 } } }
I20260923 17:28:47.553311 31996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cf83234843c40a693b6ce0211ad5e8f [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:47.553375 31995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cf83234843c40a693b6ce0211ad5e8f [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:47.553761 32003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:28:47.556819 32003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:28:47.562671 32003 catalog_manager.cc:1347] Generated new cluster ID: 5367dd2f5e9e419c84ed5db35deda61a
I20260923 17:28:47.562748 32003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:28:47.569252 32003 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:28:47.570123 32003 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:28:47.574024 32003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4cf83234843c40a693b6ce0211ad5e8f: Generated new TSK 0
I20260923 17:28:47.574679 32003 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:28:47.583943 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.1:0
--local_ip_for_outbound_sockets=127.11.229.1
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:39211
--builtin_ntp_servers=127.11.229.20:41455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 17:28:47.698908 32013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:47.699215 32013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:47.699314 32013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:47.703622 32013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:47.703696 32013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:47.703755 32013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.1
I20260923 17:28:47.709165 32013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:41455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.229.1
--webserver_port=0
--tserver_master_addrs=127.11.229.62:39211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.1
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:47.710547 32013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:47.712121 32013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:47.723112 32018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:47.723116 32021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:47.723116 32019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:47.723199 32013 server_base.cc:1034] running on GCE node
I20260923 17:28:47.723846 32013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:47.724515 32013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:47.725708 32013 hybrid_clock.cc:648] HybridClock initialized: now 1790184527725692 us; error 30 us; skew 500 ppm
I20260923 17:28:47.733768 32013 webserver.cc:460] Webserver started at http://127.11.229.1:45219/ using document root <none> and password file <none>
I20260923 17:28:47.734455 32013 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:47.734555 32013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:47.734794 32013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:47.737008 32013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-0/data/instance:
uuid: "b4d33731ea76476ca0e24061e3bf757c"
format_stamp: "Formatted at 2026-09-23 17:28:47 on dist-test-slave-ptl9"
server_key: "e56cd952d4c8288be07872ff774633f3"
server_key_iv: "221041282b145bb89ad1ba1e152003a7"
server_key_version: "encryptionkey@0"
I20260923 17:28:47.737644 32013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance:
uuid: "b4d33731ea76476ca0e24061e3bf757c"
format_stamp: "Formatted at 2026-09-23 17:28:47 on dist-test-slave-ptl9"
server_key: "e56cd952d4c8288be07872ff774633f3"
server_key_iv: "221041282b145bb89ad1ba1e152003a7"
server_key_version: "encryptionkey@0"
I20260923 17:28:47.741714 32013 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:28:47.744411 32027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:47.745700 32013 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:28:47.745837 32013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-0/wal
uuid: "b4d33731ea76476ca0e24061e3bf757c"
format_stamp: "Formatted at 2026-09-23 17:28:47 on dist-test-slave-ptl9"
server_key: "e56cd952d4c8288be07872ff774633f3"
server_key_iv: "221041282b145bb89ad1ba1e152003a7"
server_key_version: "encryptionkey@0"
I20260923 17:28:47.745963 32013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:47.767345 32013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:47.768127 32013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:47.768345 32013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:47.768992 32013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:47.770041 32013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:47.770113 32013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:47.770188 32013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:47.770244 32013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:47.779775 32013 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.1:37997
I20260923 17:28:47.779807 32140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.1:37997 every 8 connection(s)
I20260923 17:28:47.780802 32013 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-0/data/info.pb
I20260923 17:28:47.790526 31992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:47.782787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:56465 (local address 127.11.229.62:39211)
0923 17:28:47.782936 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:47.782940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:47.784013 (+ 1073us) server_negotiation.cc:407] Connection header received
0923 17:28:47.785608 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:47.785612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:47.785665 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:47.785749 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:47.786667 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:47.787236 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:47.788370 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:47.788620 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:47.788718 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:47.790028 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:47.790031 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:47.790033 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:47.790052 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:47.790073 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:47.790082 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:47.790154 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:47.790338 (+ 184us) server_negotiation.cc:299] Negotiation successful
0923 17:28:47.790399 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 17:28:47.791081 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 32013
I20260923 17:28:47.791230 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-0/wal/instance
I20260923 17:28:47.791529 12180 external_mini_cluster.cc:1428] Setting key cf46f378fee202a1ca5258d55d6c19d9
I20260923 17:28:47.791549 32143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:47.783133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:39211 (local address 127.11.229.1:56465)
0923 17:28:47.783838 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:47.783882 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:47.785387 (+ 1505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:47.785899 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:47.785905 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:47.785989 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:47.786497 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:47.786510 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:47.787352 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:47.787354 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:47.788229 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:47.788239 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:47.788839 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:47.788842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:47.789000 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:47.789729 (+ 729us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:28:47.789730 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:47.789774 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:28:47.789845 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:47.790168 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:47.790175 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:47.790180 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 17:28:47.790311 (+ 131us) client_negotiation.cc:240] Negotiation successful
0923 17:28:47.790558 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":110,"threads_started":1}
I20260923 17:28:47.793053 32141 heartbeater.cc:347] Connected to a master server at 127.11.229.62:39211
I20260923 17:28:47.793395 32141 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:47.793960 32141 heartbeater.cc:511] Master 127.11.229.62:39211 requested a full tablet report, sending...
I20260923 17:28:47.794034 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.2:0
--local_ip_for_outbound_sockets=127.11.229.2
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:39211
--builtin_ntp_servers=127.11.229.20:41455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 17:28:47.795532 31952 ts_manager.cc:194] Registered new tserver with Master: b4d33731ea76476ca0e24061e3bf757c (127.11.229.1:37997)
I20260923 17:28:47.796667 31952 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.229.1:56465
W20260923 17:28:47.908468 32144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:47.908731 32144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:47.908772 32144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:47.913051 32144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:47.913122 32144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:47.913180 32144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.2
I20260923 17:28:47.918264 32144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:41455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.229.2
--webserver_port=0
--tserver_master_addrs=127.11.229.62:39211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.2
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:47.919431 32144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:47.920889 32144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:47.931048 32149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:47.931142 32144 server_base.cc:1034] running on GCE node
W20260923 17:28:47.931048 32152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:47.931077 32150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:47.931707 32144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:47.932379 32144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:47.933565 32144 hybrid_clock.cc:648] HybridClock initialized: now 1790184527933548 us; error 32 us; skew 500 ppm
I20260923 17:28:47.935431 32144 webserver.cc:460] Webserver started at http://127.11.229.2:33581/ using document root <none> and password file <none>
I20260923 17:28:47.936035 32144 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:47.936125 32144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:47.936357 32144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:47.938584 32144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-1/data/instance:
uuid: "11e4b65fa38a4206bb0182fac5d2b1bb"
format_stamp: "Formatted at 2026-09-23 17:28:47 on dist-test-slave-ptl9"
server_key: "af1a4c2e02ab98622d48aba9c8aaa79d"
server_key_iv: "50981e1de821f4ba01fed5ccc2126096"
server_key_version: "encryptionkey@0"
I20260923 17:28:47.939168 32144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance:
uuid: "11e4b65fa38a4206bb0182fac5d2b1bb"
format_stamp: "Formatted at 2026-09-23 17:28:47 on dist-test-slave-ptl9"
server_key: "af1a4c2e02ab98622d48aba9c8aaa79d"
server_key_iv: "50981e1de821f4ba01fed5ccc2126096"
server_key_version: "encryptionkey@0"
I20260923 17:28:47.943397 32144 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:28:47.946223 32158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:47.947582 32144 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:28:47.947749 32144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-1/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-1/wal
uuid: "11e4b65fa38a4206bb0182fac5d2b1bb"
format_stamp: "Formatted at 2026-09-23 17:28:47 on dist-test-slave-ptl9"
server_key: "af1a4c2e02ab98622d48aba9c8aaa79d"
server_key_iv: "50981e1de821f4ba01fed5ccc2126096"
server_key_version: "encryptionkey@0"
I20260923 17:28:47.947898 32144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:47.977058 32144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:47.977865 32144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:47.978034 32144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:47.978596 32144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:47.979624 32144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:47.979699 32144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:47.979768 32144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:47.979810 32144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:47.989202 32144 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.2:39421
I20260923 17:28:47.989223 32271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.2:39421 every 8 connection(s)
I20260923 17:28:47.990186 32144 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-1/data/info.pb
I20260923 17:28:47.991214 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 32144
I20260923 17:28:47.991364 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-1/wal/instance
I20260923 17:28:47.991693 12180 external_mini_cluster.cc:1428] Setting key 853066042881b24807628183e2808db7
I20260923 17:28:47.993876 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.229.3:0
--local_ip_for_outbound_sockets=127.11.229.3
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:39211
--builtin_ntp_servers=127.11.229.20:41455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 17:28:48.001096 31992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:47.992201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.2:60409 (local address 127.11.229.62:39211)
0923 17:28:47.992359 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:47.992362 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:47.993707 (+ 1345us) server_negotiation.cc:407] Connection header received
0923 17:28:47.995208 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:47.995212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:47.995271 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:47.995349 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:47.996253 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:47.997142 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:47.998657 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:47.999100 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:47.999238 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:48.000452 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:48.000459 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:48.000465 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:48.000484 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:48.000511 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:48.000525 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:48.000609 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:48.000779 (+ 170us) server_negotiation.cc:299] Negotiation successful
0923 17:28:48.000866 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 17:28:48.001989 32274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:47.992722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:39211 (local address 127.11.229.2:60409)
0923 17:28:47.993558 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:47.993599 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:47.994987 (+ 1388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:47.995525 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:47.995532 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:47.995618 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:47.996112 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:47.996124 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:47.997343 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:47.997347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:47.998519 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:47.998530 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:47.999311 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:47.999314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:47.999472 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:48.000152 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:28:48.000154 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:48.000197 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:28:48.000271 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:48.000625 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:48.000632 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:48.000637 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 17:28:48.000735 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 17:28:48.000989 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":157,"threads_started":1}
I20260923 17:28:48.003353 32272 heartbeater.cc:347] Connected to a master server at 127.11.229.62:39211
I20260923 17:28:48.003652 32272 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:48.004408 32272 heartbeater.cc:511] Master 127.11.229.62:39211 requested a full tablet report, sending...
I20260923 17:28:48.005733 31952 ts_manager.cc:194] Registered new tserver with Master: 11e4b65fa38a4206bb0182fac5d2b1bb (127.11.229.2:39421)
I20260923 17:28:48.006351 31952 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.229.2:60409
W20260923 17:28:48.112673 32275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:48.112982 32275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:48.113029 32275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:48.117249 32275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:48.117352 32275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:48.117410 32275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.229.3
I20260923 17:28:48.122660 32275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:41455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.229.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.229.3
--webserver_port=0
--tserver_master_addrs=127.11.229.62:39211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.229.3
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:48.123873 32275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:48.125394 32275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:48.135221 32280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:48.135236 32281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:48.135237 32283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:48.135332 32275 server_base.cc:1034] running on GCE node
I20260923 17:28:48.135950 32275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:48.136615 32275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:48.137832 32275 hybrid_clock.cc:648] HybridClock initialized: now 1790184528137793 us; error 49 us; skew 500 ppm
I20260923 17:28:48.139720 32275 webserver.cc:460] Webserver started at http://127.11.229.3:45485/ using document root <none> and password file <none>
I20260923 17:28:48.140326 32275 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:48.140421 32275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:48.140656 32275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:48.142802 32275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-2/data/instance:
uuid: "c965a678857d42d79a457c180ce5abc8"
format_stamp: "Formatted at 2026-09-23 17:28:48 on dist-test-slave-ptl9"
server_key: "c67d348f787d412e6bf607d02deb7110"
server_key_iv: "480deb7cecb79a403d95505838a18edb"
server_key_version: "encryptionkey@0"
I20260923 17:28:48.143428 32275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance:
uuid: "c965a678857d42d79a457c180ce5abc8"
format_stamp: "Formatted at 2026-09-23 17:28:48 on dist-test-slave-ptl9"
server_key: "c67d348f787d412e6bf607d02deb7110"
server_key_iv: "480deb7cecb79a403d95505838a18edb"
server_key_version: "encryptionkey@0"
I20260923 17:28:48.147164 32275 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 17:28:48.149816 32289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:48.151273 32275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:28:48.151412 32275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-2/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-2/wal
uuid: "c965a678857d42d79a457c180ce5abc8"
format_stamp: "Formatted at 2026-09-23 17:28:48 on dist-test-slave-ptl9"
server_key: "c67d348f787d412e6bf607d02deb7110"
server_key_iv: "480deb7cecb79a403d95505838a18edb"
server_key_version: "encryptionkey@0"
I20260923 17:28:48.151535 32275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:48.180840 32275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:48.181666 32275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:48.181878 32275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:48.182526 32275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:48.183539 32275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:48.183614 32275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:48.183687 32275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:48.183739 32275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:48.193053 32275 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.3:41021
I20260923 17:28:48.193075 32402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.3:41021 every 8 connection(s)
I20260923 17:28:48.194082 32275 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-2/data/info.pb
I20260923 17:28:48.201736 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 32275
I20260923 17:28:48.201864 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-2/wal/instance
I20260923 17:28:48.202194 12180 external_mini_cluster.cc:1428] Setting key ec571ea552576b0441dc2dfa07c15b3a
I20260923 17:28:48.203912 31992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:48.195977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.3:49901 (local address 127.11.229.62:39211)
0923 17:28:48.196158 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:48.196161 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:48.197234 (+ 1073us) server_negotiation.cc:407] Connection header received
0923 17:28:48.198757 (+ 1523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:48.198760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:48.198813 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:48.198908 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:48.199865 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.200412 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.201635 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.201991 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.202117 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:48.203419 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:48.203425 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:48.203428 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:48.203448 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:48.203468 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:48.203477 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:48.203540 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:48.203732 (+ 192us) server_negotiation.cc:299] Negotiation successful
0923 17:28:48.203798 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 17:28:48.204969 32405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:48.196363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:39211 (local address 127.11.229.3:49901)
0923 17:28:48.197080 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:48.197122 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:48.198492 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:48.199091 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:48.199098 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:48.199185 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:48.199694 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:48.199707 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.200528 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.200531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:48.201507 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:48.201517 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.202196 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.202199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:48.202429 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:48.203075 (+ 646us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:28:48.203097 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:48.203155 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:28:48.203246 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:48.203572 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:48.203581 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:48.203588 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 17:28:48.203701 (+ 113us) client_negotiation.cc:240] Negotiation successful
0923 17:28:48.204010 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":106,"threads_started":1}
I20260923 17:28:48.206149 32403 heartbeater.cc:347] Connected to a master server at 127.11.229.62:39211
I20260923 17:28:48.206416 32403 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:48.206967 32403 heartbeater.cc:511] Master 127.11.229.62:39211 requested a full tablet report, sending...
I20260923 17:28:48.208073 31952 ts_manager.cc:194] Registered new tserver with Master: c965a678857d42d79a457c180ce5abc8 (127.11.229.3:41021)
I20260923 17:28:48.208647 31952 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.229.3:49901
I20260923 17:28:48.216272 12180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:28:48.222074 31992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:48.218362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55246 (local address 127.11.229.62:39211)
0923 17:28:48.218490 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:48.218494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:48.218720 (+ 226us) server_negotiation.cc:407] Connection header received
0923 17:28:48.218827 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:48.218829 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:48.218870 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:48.218941 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:48.219207 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.219740 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.220586 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.220792 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.220867 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:48.221719 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:48.221722 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:48.221724 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:48.221736 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:48.221756 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:48.221764 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:48.221829 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:48.221903 (+ 74us) server_negotiation.cc:299] Negotiation successful
0923 17:28:48.221954 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 17:28:48.225039 31952 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:28:48.228451 31952 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:28:48.243427 32418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:48.238373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35188 (local address 127.11.229.1:37997)
0923 17:28:48.238782 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:48.238793 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:48.239036 (+ 243us) server_negotiation.cc:407] Connection header received
0923 17:28:48.239136 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:48.239140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:48.239319 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:48.239470 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:48.239777 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.240594 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.242199 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.242714 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.242852 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:48.243056 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:48.243132 (+ 76us) server_negotiation.cc:299] Negotiation successful
0923 17:28:48.243278 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":88,"threads_started":1}
I20260923 17:28:48.243575 32414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:48.237424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:41021 (local address 127.0.0.1:33354)
0923 17:28:48.237805 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:48.237852 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:48.237993 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:48.239402 (+ 1409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:48.239405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:48.239423 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:48.239508 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:48.239516 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.240843 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.240846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:48.241925 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:48.241936 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.243013 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.243016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:48.243223 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:48.243289 (+ 66us) client_negotiation.cc:769] Sending connection context
0923 17:28:48.243342 (+ 53us) client_negotiation.cc:240] Negotiation successful
0923 17:28:48.243409 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":2,"spinlock_wait_cycles":6272,"thread_start_us":122,"threads_started":1}
I20260923 17:28:48.243403 32417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:48.237970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:37997 (local address 127.0.0.1:35188)
0923 17:28:48.238838 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:48.238851 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:48.238936 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:48.239577 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:48.239581 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:48.239599 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:48.239668 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:48.239672 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.240721 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.240724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:48.242047 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:48.242058 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.242848 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.242852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:48.242955 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:48.242991 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 17:28:48.243104 (+ 113us) client_negotiation.cc:240] Negotiation successful
0923 17:28:48.243228 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"mutex_wait_us":77,"thread_start_us":85,"threads_started":1}
I20260923 17:28:48.243845 32415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:48.237211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:39421 (local address 127.0.0.1:51152)
0923 17:28:48.237801 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:48.237852 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:48.237993 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:48.238837 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:48.238845 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:48.238871 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:48.239016 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:48.239024 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.240453 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.240456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:48.242047 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:48.242058 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.243243 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.243247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:48.243382 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:48.243402 (+ 20us) client_negotiation.cc:769] Sending connection context
0923 17:28:48.243466 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 17:28:48.243534 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":114,"spinlock_wait_cycles":5504,"thread_start_us":130,"threads_started":1}
I20260923 17:28:48.244148 32416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:48.237433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51152 (local address 127.11.229.2:39421)
0923 17:28:48.238291 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:48.238305 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:48.238343 (+ 38us) server_negotiation.cc:407] Connection header received
0923 17:28:48.238426 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:48.238431 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:48.238648 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:48.238822 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:48.239154 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.240321 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.242178 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.242903 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.243002 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:48.243184 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:48.243641 (+ 457us) server_negotiation.cc:299] Negotiation successful
0923 17:28:48.243737 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":752,"thread_start_us":112,"threads_started":1}
I20260923 17:28:48.244504 32419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:48.238426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33354 (local address 127.11.229.3:41021)
0923 17:28:48.238991 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:48.239002 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:48.239024 (+ 22us) server_negotiation.cc:407] Connection header received
0923 17:28:48.239099 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:48.239107 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:48.239268 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:48.239426 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:48.239629 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.240698 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.242058 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.242868 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.244034 (+ 1166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:48.244242 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:48.244296 (+ 54us) server_negotiation.cc:299] Negotiation successful
0923 17:28:48.244386 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":149,"threads_started":1}
I20260923 17:28:48.246908 32075 tablet_service.cc:1467] Processing CreateTablet for tablet adc59f72eacf42c9b1c499300edf5e64 (DEFAULT_TABLE table=test-table [id=fce960c819a54b7ab1a05639b75faa47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:48.247377 32206 tablet_service.cc:1467] Processing CreateTablet for tablet adc59f72eacf42c9b1c499300edf5e64 (DEFAULT_TABLE table=test-table [id=fce960c819a54b7ab1a05639b75faa47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:48.247377 32337 tablet_service.cc:1467] Processing CreateTablet for tablet adc59f72eacf42c9b1c499300edf5e64 (DEFAULT_TABLE table=test-table [id=fce960c819a54b7ab1a05639b75faa47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:48.248059 32075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adc59f72eacf42c9b1c499300edf5e64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:48.248405 32206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adc59f72eacf42c9b1c499300edf5e64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:48.248404 32337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adc59f72eacf42c9b1c499300edf5e64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:48.255728 32420 tablet_bootstrap.cc:492] T adc59f72eacf42c9b1c499300edf5e64 P b4d33731ea76476ca0e24061e3bf757c: Bootstrap starting.
I20260923 17:28:48.256423 32421 tablet_bootstrap.cc:492] T adc59f72eacf42c9b1c499300edf5e64 P c965a678857d42d79a457c180ce5abc8: Bootstrap starting.
I20260923 17:28:48.258728 32421 tablet_bootstrap.cc:654] T adc59f72eacf42c9b1c499300edf5e64 P c965a678857d42d79a457c180ce5abc8: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:48.258729 32420 tablet_bootstrap.cc:654] T adc59f72eacf42c9b1c499300edf5e64 P b4d33731ea76476ca0e24061e3bf757c: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:48.259047 32422 tablet_bootstrap.cc:492] T adc59f72eacf42c9b1c499300edf5e64 P 11e4b65fa38a4206bb0182fac5d2b1bb: Bootstrap starting.
I20260923 17:28:48.259667 32421 log.cc:824] T adc59f72eacf42c9b1c499300edf5e64 P c965a678857d42d79a457c180ce5abc8: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:48.260524 32420 log.cc:824] T adc59f72eacf42c9b1c499300edf5e64 P b4d33731ea76476ca0e24061e3bf757c: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:48.261421 32422 tablet_bootstrap.cc:654] T adc59f72eacf42c9b1c499300edf5e64 P 11e4b65fa38a4206bb0182fac5d2b1bb: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:48.262092 32421 tablet_bootstrap.cc:492] T adc59f72eacf42c9b1c499300edf5e64 P c965a678857d42d79a457c180ce5abc8: No bootstrap required, opened a new log
I20260923 17:28:48.262118 32422 log.cc:824] T adc59f72eacf42c9b1c499300edf5e64 P 11e4b65fa38a4206bb0182fac5d2b1bb: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:48.262303 32421 ts_tablet_manager.cc:1397] T adc59f72eacf42c9b1c499300edf5e64 P c965a678857d42d79a457c180ce5abc8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 17:28:48.262817 32420 tablet_bootstrap.cc:492] T adc59f72eacf42c9b1c499300edf5e64 P b4d33731ea76476ca0e24061e3bf757c: No bootstrap required, opened a new log
I20260923 17:28:48.263041 32420 ts_tablet_manager.cc:1397] T adc59f72eacf42c9b1c499300edf5e64 P b4d33731ea76476ca0e24061e3bf757c: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260923 17:28:48.263872 32422 tablet_bootstrap.cc:492] T adc59f72eacf42c9b1c499300edf5e64 P 11e4b65fa38a4206bb0182fac5d2b1bb: No bootstrap required, opened a new log
I20260923 17:28:48.264082 32422 ts_tablet_manager.cc:1397] T adc59f72eacf42c9b1c499300edf5e64 P 11e4b65fa38a4206bb0182fac5d2b1bb: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260923 17:28:48.266956 32421 raft_consensus.cc:348] T adc59f72eacf42c9b1c499300edf5e64 P c965a678857d42d79a457c180ce5abc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e4b65fa38a4206bb0182fac5d2b1bb" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39421 } } peers { permanent_uuid: "c965a678857d42d79a457c180ce5abc8" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41021 } } peers { permanent_uuid: "b4d33731ea76476ca0e24061e3bf757c" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 37997 } }
I20260923 17:28:48.267261 32421 raft_consensus.cc:374] T adc59f72eacf42c9b1c499300edf5e64 P c965a678857d42d79a457c180ce5abc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:48.267313 32421 raft_consensus.cc:729] T adc59f72eacf42c9b1c499300edf5e64 P c965a678857d42d79a457c180ce5abc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c965a678857d42d79a457c180ce5abc8, State: Initialized, Role: FOLLOWER
I20260923 17:28:48.267556 32422 raft_consensus.cc:348] T adc59f72eacf42c9b1c499300edf5e64 P 11e4b65fa38a4206bb0182fac5d2b1bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e4b65fa38a4206bb0182fac5d2b1bb" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39421 } } peers { permanent_uuid: "c965a678857d42d79a457c180ce5abc8" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41021 } } peers { permanent_uuid: "b4d33731ea76476ca0e24061e3bf757c" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 37997 } }
I20260923 17:28:48.267616 32420 raft_consensus.cc:348] T adc59f72eacf42c9b1c499300edf5e64 P b4d33731ea76476ca0e24061e3bf757c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e4b65fa38a4206bb0182fac5d2b1bb" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39421 } } peers { permanent_uuid: "c965a678857d42d79a457c180ce5abc8" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41021 } } peers { permanent_uuid: "b4d33731ea76476ca0e24061e3bf757c" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 37997 } }
I20260923 17:28:48.267840 32422 raft_consensus.cc:374] T adc59f72eacf42c9b1c499300edf5e64 P 11e4b65fa38a4206bb0182fac5d2b1bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:48.267859 32420 raft_consensus.cc:374] T adc59f72eacf42c9b1c499300edf5e64 P b4d33731ea76476ca0e24061e3bf757c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:48.267807 32421 consensus_queue.cc:260] T adc59f72eacf42c9b1c499300edf5e64 P c965a678857d42d79a457c180ce5abc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e4b65fa38a4206bb0182fac5d2b1bb" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39421 } } peers { permanent_uuid: "c965a678857d42d79a457c180ce5abc8" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41021 } } peers { permanent_uuid: "b4d33731ea76476ca0e24061e3bf757c" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 37997 } }
I20260923 17:28:48.267904 32422 raft_consensus.cc:729] T adc59f72eacf42c9b1c499300edf5e64 P 11e4b65fa38a4206bb0182fac5d2b1bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11e4b65fa38a4206bb0182fac5d2b1bb, State: Initialized, Role: FOLLOWER
I20260923 17:28:48.267917 32420 raft_consensus.cc:729] T adc59f72eacf42c9b1c499300edf5e64 P b4d33731ea76476ca0e24061e3bf757c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4d33731ea76476ca0e24061e3bf757c, State: Initialized, Role: FOLLOWER
I20260923 17:28:48.268345 32420 consensus_queue.cc:260] T adc59f72eacf42c9b1c499300edf5e64 P b4d33731ea76476ca0e24061e3bf757c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e4b65fa38a4206bb0182fac5d2b1bb" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39421 } } peers { permanent_uuid: "c965a678857d42d79a457c180ce5abc8" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41021 } } peers { permanent_uuid: "b4d33731ea76476ca0e24061e3bf757c" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 37997 } }
I20260923 17:28:48.268383 32422 consensus_queue.cc:260] T adc59f72eacf42c9b1c499300edf5e64 P 11e4b65fa38a4206bb0182fac5d2b1bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e4b65fa38a4206bb0182fac5d2b1bb" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39421 } } peers { permanent_uuid: "c965a678857d42d79a457c180ce5abc8" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41021 } } peers { permanent_uuid: "b4d33731ea76476ca0e24061e3bf757c" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 37997 } }
I20260923 17:28:48.268970 32421 ts_tablet_manager.cc:1428] T adc59f72eacf42c9b1c499300edf5e64 P c965a678857d42d79a457c180ce5abc8: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260923 17:28:48.269079 32141 heartbeater.cc:503] Master 127.11.229.62:39211 was elected leader, sending a full tablet report...
I20260923 17:28:48.269342 32420 ts_tablet_manager.cc:1428] T adc59f72eacf42c9b1c499300edf5e64 P b4d33731ea76476ca0e24061e3bf757c: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260923 17:28:48.269347 32403 heartbeater.cc:503] Master 127.11.229.62:39211 was elected leader, sending a full tablet report...
I20260923 17:28:48.269408 32422 ts_tablet_manager.cc:1428] T adc59f72eacf42c9b1c499300edf5e64 P 11e4b65fa38a4206bb0182fac5d2b1bb: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260923 17:28:48.269702 32272 heartbeater.cc:503] Master 127.11.229.62:39211 was elected leader, sending a full tablet report...
W20260923 17:28:48.282124 32142 tablet.cc:2369] T adc59f72eacf42c9b1c499300edf5e64 P b4d33731ea76476ca0e24061e3bf757c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:28:48.379873 32427 raft_consensus.cc:482] T adc59f72eacf42c9b1c499300edf5e64 P b4d33731ea76476ca0e24061e3bf757c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:28:48.380107 32427 raft_consensus.cc:504] T adc59f72eacf42c9b1c499300edf5e64 P b4d33731ea76476ca0e24061e3bf757c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e4b65fa38a4206bb0182fac5d2b1bb" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39421 } } peers { permanent_uuid: "c965a678857d42d79a457c180ce5abc8" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41021 } } peers { permanent_uuid: "b4d33731ea76476ca0e24061e3bf757c" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 37997 } }
I20260923 17:28:48.381350 32427 leader_election.cc:288] T adc59f72eacf42c9b1c499300edf5e64 P b4d33731ea76476ca0e24061e3bf757c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11e4b65fa38a4206bb0182fac5d2b1bb (127.11.229.2:39421), c965a678857d42d79a457c180ce5abc8 (127.11.229.3:41021)
I20260923 17:28:48.386897 32419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:48.381679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:52427 (local address 127.11.229.3:41021)
0923 17:28:48.381812 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:48.381818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:48.382127 (+ 309us) server_negotiation.cc:407] Connection header received
0923 17:28:48.382279 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:48.382282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:48.382332 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:48.382424 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:48.382902 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.383694 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.385535 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.386152 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.386285 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:48.386336 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:48.386683 (+ 347us) server_negotiation.cc:299] Negotiation successful
0923 17:28:48.386757 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 17:28:48.386900 32416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:48.381716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.229.1:37835 (local address 127.11.229.2:39421)
0923 17:28:48.381852 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:48.381856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:48.382191 (+ 335us) server_negotiation.cc:407] Connection header received
0923 17:28:48.382274 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:48.382277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:48.382332 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:48.382423 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:48.382909 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.383692 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.385535 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.386153 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.386285 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:48.386336 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:48.386683 (+ 347us) server_negotiation.cc:299] Negotiation successful
0923 17:28:48.386757 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 17:28:48.386965 32429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:48.381615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.2:39421 (local address 127.11.229.1:37835)
0923 17:28:48.382099 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:48.382111 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:48.382183 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:48.382460 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:48.382463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:48.382504 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:48.382772 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:48.382779 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.383820 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.383823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:48.385394 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:48.385407 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.386296 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.386300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:48.386505 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:48.386568 (+ 63us) client_negotiation.cc:769] Sending connection context
0923 17:28:48.386649 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 17:28:48.386768 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":292,"spinlock_wait_cycles":30208,"thread_start_us":119,"threads_started":1}
I20260923 17:28:48.386965 32430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:48.381583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.3:41021 (local address 127.11.229.1:52427)
0923 17:28:48.382006 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:48.382027 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:48.382162 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:48.382460 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:48.382463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:48.382506 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:48.382772 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:48.382779 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.383814 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.383818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:48.385394 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:48.385407 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.386296 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.386300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:48.386489 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:48.386573 (+ 84us) client_negotiation.cc:769] Sending connection context
0923 17:28:48.386649 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 17:28:48.386766 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":348,"spinlock_wait_cycles":6528,"thread_start_us":103,"threads_started":1}
I20260923 17:28:48.387694 32357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adc59f72eacf42c9b1c499300edf5e64" candidate_uuid: "b4d33731ea76476ca0e24061e3bf757c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c965a678857d42d79a457c180ce5abc8" is_pre_election: true
I20260923 17:28:48.387926 32226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adc59f72eacf42c9b1c499300edf5e64" candidate_uuid: "b4d33731ea76476ca0e24061e3bf757c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11e4b65fa38a4206bb0182fac5d2b1bb" is_pre_election: true
I20260923 17:28:48.388055 32357 raft_consensus.cc:2454] T adc59f72eacf42c9b1c499300edf5e64 P c965a678857d42d79a457c180ce5abc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4d33731ea76476ca0e24061e3bf757c in term 0.
I20260923 17:28:48.388177 32226 raft_consensus.cc:2454] T adc59f72eacf42c9b1c499300edf5e64 P 11e4b65fa38a4206bb0182fac5d2b1bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4d33731ea76476ca0e24061e3bf757c in term 0.
I20260923 17:28:48.388664 32031 leader_election.cc:302] T adc59f72eacf42c9b1c499300edf5e64 P b4d33731ea76476ca0e24061e3bf757c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 11e4b65fa38a4206bb0182fac5d2b1bb, b4d33731ea76476ca0e24061e3bf757c, c965a678857d42d79a457c180ce5abc8; no voters:
I20260923 17:28:48.388975 32427 raft_consensus.cc:2781] T adc59f72eacf42c9b1c499300edf5e64 P b4d33731ea76476ca0e24061e3bf757c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:28:48.389079 32427 raft_consensus.cc:482] T adc59f72eacf42c9b1c499300edf5e64 P b4d33731ea76476ca0e24061e3bf757c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:28:48.389134 32427 raft_consensus.cc:3037] T adc59f72eacf42c9b1c499300edf5e64 P b4d33731ea76476ca0e24061e3bf757c [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:48.390340 32427 raft_consensus.cc:504] T adc59f72eacf42c9b1c499300edf5e64 P b4d33731ea76476ca0e24061e3bf757c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e4b65fa38a4206bb0182fac5d2b1bb" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39421 } } peers { permanent_uuid: "c965a678857d42d79a457c180ce5abc8" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41021 } } peers { permanent_uuid: "b4d33731ea76476ca0e24061e3bf757c" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 37997 } }
I20260923 17:28:48.390731 32427 leader_election.cc:288] T adc59f72eacf42c9b1c499300edf5e64 P b4d33731ea76476ca0e24061e3bf757c [CANDIDATE]: Term 1 election: Requested vote from peers 11e4b65fa38a4206bb0182fac5d2b1bb (127.11.229.2:39421), c965a678857d42d79a457c180ce5abc8 (127.11.229.3:41021)
I20260923 17:28:48.391074 32226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adc59f72eacf42c9b1c499300edf5e64" candidate_uuid: "b4d33731ea76476ca0e24061e3bf757c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11e4b65fa38a4206bb0182fac5d2b1bb"
I20260923 17:28:48.391113 32357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adc59f72eacf42c9b1c499300edf5e64" candidate_uuid: "b4d33731ea76476ca0e24061e3bf757c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c965a678857d42d79a457c180ce5abc8"
I20260923 17:28:48.391185 32226 raft_consensus.cc:3037] T adc59f72eacf42c9b1c499300edf5e64 P 11e4b65fa38a4206bb0182fac5d2b1bb [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:48.391268 32357 raft_consensus.cc:3037] T adc59f72eacf42c9b1c499300edf5e64 P c965a678857d42d79a457c180ce5abc8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:48.392292 32226 raft_consensus.cc:2454] T adc59f72eacf42c9b1c499300edf5e64 P 11e4b65fa38a4206bb0182fac5d2b1bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4d33731ea76476ca0e24061e3bf757c in term 1.
I20260923 17:28:48.392409 32357 raft_consensus.cc:2454] T adc59f72eacf42c9b1c499300edf5e64 P c965a678857d42d79a457c180ce5abc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4d33731ea76476ca0e24061e3bf757c in term 1.
I20260923 17:28:48.392643 32028 leader_election.cc:302] T adc59f72eacf42c9b1c499300edf5e64 P b4d33731ea76476ca0e24061e3bf757c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11e4b65fa38a4206bb0182fac5d2b1bb, b4d33731ea76476ca0e24061e3bf757c; no voters:
I20260923 17:28:48.392870 32427 raft_consensus.cc:2781] T adc59f72eacf42c9b1c499300edf5e64 P b4d33731ea76476ca0e24061e3bf757c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:48.393136 32427 raft_consensus.cc:686] T adc59f72eacf42c9b1c499300edf5e64 P b4d33731ea76476ca0e24061e3bf757c [term 1 LEADER]: Becoming Leader. State: Replica: b4d33731ea76476ca0e24061e3bf757c, State: Running, Role: LEADER
I20260923 17:28:48.393563 32427 consensus_queue.cc:237] T adc59f72eacf42c9b1c499300edf5e64 P b4d33731ea76476ca0e24061e3bf757c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e4b65fa38a4206bb0182fac5d2b1bb" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39421 } } peers { permanent_uuid: "c965a678857d42d79a457c180ce5abc8" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41021 } } peers { permanent_uuid: "b4d33731ea76476ca0e24061e3bf757c" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 37997 } }
I20260923 17:28:48.396880 31953 catalog_manager.cc:5495] T adc59f72eacf42c9b1c499300edf5e64 P b4d33731ea76476ca0e24061e3bf757c reported cstate change: term changed from 0 to 1, leader changed from <none> to b4d33731ea76476ca0e24061e3bf757c (127.11.229.1). New cstate: current_term: 1 leader_uuid: "b4d33731ea76476ca0e24061e3bf757c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e4b65fa38a4206bb0182fac5d2b1bb" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c965a678857d42d79a457c180ce5abc8" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4d33731ea76476ca0e24061e3bf757c" member_type: VOTER last_known_addr { host: "127.11.229.1" port: 37997 } health_report { overall_health: HEALTHY } } }
I20260923 17:28:48.435173 32418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:48.431403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35204 (local address 127.11.229.1:37997)
0923 17:28:48.431603 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:48.431608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:48.431625 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:28:48.431716 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:48.431718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:48.431778 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:48.431910 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:28:48.432175 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.432684 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.433638 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.433864 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.433978 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:48.434304 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:28:48.434746 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:28:48.434902 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:48.434962 (+ 60us) server_negotiation.cc:299] Negotiation successful
0923 17:28:48.435019 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 17:28:48.441569 32357 raft_consensus.cc:1261] T adc59f72eacf42c9b1c499300edf5e64 P c965a678857d42d79a457c180ce5abc8 [term 1 FOLLOWER]: Refusing update from remote peer b4d33731ea76476ca0e24061e3bf757c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:28:48.441569 32226 raft_consensus.cc:1261] T adc59f72eacf42c9b1c499300edf5e64 P 11e4b65fa38a4206bb0182fac5d2b1bb [term 1 FOLLOWER]: Refusing update from remote peer b4d33731ea76476ca0e24061e3bf757c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:28:48.442327 32431 consensus_queue.cc:1035] T adc59f72eacf42c9b1c499300edf5e64 P b4d33731ea76476ca0e24061e3bf757c [LEADER]: Connected to new peer: Peer: permanent_uuid: "11e4b65fa38a4206bb0182fac5d2b1bb" member_type: VOTER last_known_addr { host: "127.11.229.2" port: 39421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:28:48.442570 32427 consensus_queue.cc:1035] T adc59f72eacf42c9b1c499300edf5e64 P b4d33731ea76476ca0e24061e3bf757c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c965a678857d42d79a457c180ce5abc8" member_type: VOTER last_known_addr { host: "127.11.229.3" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 17:28:48.445030 32404 tablet.cc:2369] T adc59f72eacf42c9b1c499300edf5e64 P c965a678857d42d79a457c180ce5abc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:28:48.465215 31951 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:28:48.465479 31951 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:28:48.469337 31951 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4cf83234843c40a693b6ce0211ad5e8f: Sending DeleteTablet for 3 replicas of tablet adc59f72eacf42c9b1c499300edf5e64
I20260923 17:28:48.470232 32206 tablet_service.cc:1514] Processing DeleteTablet for tablet adc59f72eacf42c9b1c499300edf5e64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:48 UTC) from {username='slave'} at 127.0.0.1:51152
I20260923 17:28:48.470713 32337 tablet_service.cc:1514] Processing DeleteTablet for tablet adc59f72eacf42c9b1c499300edf5e64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:48 UTC) from {username='slave'} at 127.0.0.1:33354
I20260923 17:28:48.470884 32075 tablet_service.cc:1514] Processing DeleteTablet for tablet adc59f72eacf42c9b1c499300edf5e64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:48 UTC) from {username='slave'} at 127.0.0.1:35188
I20260923 17:28:48.471256 32444 tablet_replica.cc:330] T adc59f72eacf42c9b1c499300edf5e64 P c965a678857d42d79a457c180ce5abc8: stopping tablet replica
I20260923 17:28:48.471400 32445 tablet_replica.cc:330] T adc59f72eacf42c9b1c499300edf5e64 P b4d33731ea76476ca0e24061e3bf757c: stopping tablet replica
I20260923 17:28:48.471732 32444 raft_consensus.cc:2229] T adc59f72eacf42c9b1c499300edf5e64 P c965a678857d42d79a457c180ce5abc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:28:48.471979 32443 tablet_replica.cc:330] T adc59f72eacf42c9b1c499300edf5e64 P 11e4b65fa38a4206bb0182fac5d2b1bb: stopping tablet replica
I20260923 17:28:48.472225 32444 raft_consensus.cc:2258] T adc59f72eacf42c9b1c499300edf5e64 P c965a678857d42d79a457c180ce5abc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:48.472272 32443 raft_consensus.cc:2229] T adc59f72eacf42c9b1c499300edf5e64 P 11e4b65fa38a4206bb0182fac5d2b1bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:28:48.472319 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 32013
I20260923 17:28:48.472574 32443 raft_consensus.cc:2258] T adc59f72eacf42c9b1c499300edf5e64 P 11e4b65fa38a4206bb0182fac5d2b1bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:48.474639 32444 ts_tablet_manager.cc:1905] T adc59f72eacf42c9b1c499300edf5e64 P c965a678857d42d79a457c180ce5abc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:48.475106 32443 ts_tablet_manager.cc:1905] T adc59f72eacf42c9b1c499300edf5e64 P 11e4b65fa38a4206bb0182fac5d2b1bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:48.477698 32444 ts_tablet_manager.cc:1918] T adc59f72eacf42c9b1c499300edf5e64 P c965a678857d42d79a457c180ce5abc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:28:48.477698 32443 ts_tablet_manager.cc:1918] T adc59f72eacf42c9b1c499300edf5e64 P 11e4b65fa38a4206bb0182fac5d2b1bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:28:48.477803 32443 log.cc:1196] T adc59f72eacf42c9b1c499300edf5e64 P 11e4b65fa38a4206bb0182fac5d2b1bb: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-1/wal/wals/adc59f72eacf42c9b1c499300edf5e64
I20260923 17:28:48.477937 32444 log.cc:1196] T adc59f72eacf42c9b1c499300edf5e64 P c965a678857d42d79a457c180ce5abc8: Deleting WAL directory at /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184271597484-12180-0/minicluster-data/ts-2/wal/wals/adc59f72eacf42c9b1c499300edf5e64
I20260923 17:28:48.478139 32443 ts_tablet_manager.cc:1939] T adc59f72eacf42c9b1c499300edf5e64 P 11e4b65fa38a4206bb0182fac5d2b1bb: Deleting consensus metadata
I20260923 17:28:48.478363 32444 ts_tablet_manager.cc:1939] T adc59f72eacf42c9b1c499300edf5e64 P c965a678857d42d79a457c180ce5abc8: Deleting consensus metadata
I20260923 17:28:48.479513 31940 catalog_manager.cc:4841] TS c965a678857d42d79a457c180ce5abc8 (127.11.229.3:41021): tablet adc59f72eacf42c9b1c499300edf5e64 (table test-table [id=fce960c819a54b7ab1a05639b75faa47]) successfully deleted
I20260923 17:28:48.479607 31937 catalog_manager.cc:4841] TS 11e4b65fa38a4206bb0182fac5d2b1bb (127.11.229.2:39421): tablet adc59f72eacf42c9b1c499300edf5e64 (table test-table [id=fce960c819a54b7ab1a05639b75faa47]) successfully deleted
W20260923 17:28:48.480233 31940 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.229.1:37997 (error 108)
I20260923 17:28:48.481010 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 32144
I20260923 17:28:48.481308 32415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:48.480972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.1:37997 (local address 127.0.0.1:35220)
0923 17:28:48.481110 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:48.481183 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.1:37997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260923 17:28:48.482506 31940 catalog_manager.cc:4584] TS b4d33731ea76476ca0e24061e3bf757c (127.11.229.1:37997): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet adc59f72eacf42c9b1c499300edf5e64: Network error: Client connection negotiation failed: client connection to 127.11.229.1:37997: connect: Connection refused (error 111)
I20260923 17:28:48.487213 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 32275
I20260923 17:28:48.493461 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 31922
2026-09-23T17:28:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1196 ms)
[----------] 6 tests from EncryptionPolicyTest (11030 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:28:48 dist-test-slave-ptl9 krb5kdc[32449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:28:48 dist-test-slave-ptl9 krb5kdc[32449](info): set up 2 sockets
Sep 23 17:28:48 dist-test-slave-ptl9 krb5kdc[32449](info): commencing operation
krb5kdc: starting...
W20260923 17:28:49.549737 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:28:49 dist-test-slave-ptl9 krb5kdc[32449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:28:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:28:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:28:49.677495 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40777
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:49.791062 32462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:49.791376 32462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:49.791456 32462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:49.795501 32462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:28:49.795600 32462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:49.795640 32462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:49.795699 32462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:28:49.795737 32462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:28:49.801018 32462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40777
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:49.802201 32462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:49.803258 32462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:49.812084 32468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:49.812088 32470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:49.812100 32467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:49.812453 32462 server_base.cc:1034] running on GCE node
I20260923 17:28:49.813068 32462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:49.814405 32462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:49.815642 32462 hybrid_clock.cc:648] HybridClock initialized: now 1790184529815624 us; error 31 us; skew 500 ppm
Sep 23 17:28:49 dist-test-slave-ptl9 krb5kdc[32449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:49.818607 32462 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:28:49.819536 32462 webserver.cc:460] Webserver started at http://127.0.0.1:46155/ using document root <none> and password file <none>
I20260923 17:28:49.820128 32462 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:49.820220 32462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:49.820462 32462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:49.822703 32462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "ad6915b950994d6483441acad6640127"
format_stamp: "Formatted at 2026-09-23 17:28:49 on dist-test-slave-ptl9"
server_key: "1e059580432f0878260f791385b2c143"
server_key_iv: "0ee60cd11c07fd04fb786b97fa9665b2"
server_key_version: "encryptionkey@0"
I20260923 17:28:49.823313 32462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "ad6915b950994d6483441acad6640127"
format_stamp: "Formatted at 2026-09-23 17:28:49 on dist-test-slave-ptl9"
server_key: "1e059580432f0878260f791385b2c143"
server_key_iv: "0ee60cd11c07fd04fb786b97fa9665b2"
server_key_version: "encryptionkey@0"
I20260923 17:28:49.827291 32462 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:28:49.829797 32477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:49.831037 32462 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:28:49.831187 32462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "ad6915b950994d6483441acad6640127"
format_stamp: "Formatted at 2026-09-23 17:28:49 on dist-test-slave-ptl9"
server_key: "1e059580432f0878260f791385b2c143"
server_key_iv: "0ee60cd11c07fd04fb786b97fa9665b2"
server_key_version: "encryptionkey@0"
I20260923 17:28:49.831316 32462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:49.847751 32462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:49.851271 32462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:49.851492 32462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:49.858865 32462 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40777
I20260923 17:28:49.858888 32529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40777 every 8 connection(s)
I20260923 17:28:49.859869 32462 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:28:49.862882 32530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:49.864427 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 32462
I20260923 17:28:49.864563 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:28:49.864912 12180 external_mini_cluster.cc:1428] Setting key 342fbfaa690522520c255339af98eb69
I20260923 17:28:49.869728 32530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad6915b950994d6483441acad6640127: Bootstrap starting.
I20260923 17:28:49.872411 32530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad6915b950994d6483441acad6640127: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:49.873219 32530 log.cc:824] T 00000000000000000000000000000000 P ad6915b950994d6483441acad6640127: Log is configured to *not* fsync() on all Append() calls
Sep 23 17:28:49 dist-test-slave-ptl9 krb5kdc[32449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 17:28:49.875597 32530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad6915b950994d6483441acad6640127: No bootstrap required, opened a new log
I20260923 17:28:49.878965 32530 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad6915b950994d6483441acad6640127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad6915b950994d6483441acad6640127" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40777 } }
I20260923 17:28:49.879243 32530 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad6915b950994d6483441acad6640127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:49.879307 32530 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad6915b950994d6483441acad6640127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad6915b950994d6483441acad6640127, State: Initialized, Role: FOLLOWER
I20260923 17:28:49.879868 32530 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad6915b950994d6483441acad6640127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad6915b950994d6483441acad6640127" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40777 } }
I20260923 17:28:49.880025 32530 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad6915b950994d6483441acad6640127 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:28:49.880092 32530 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad6915b950994d6483441acad6640127 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:28:49.880203 32530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad6915b950994d6483441acad6640127 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:49.881340 32530 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad6915b950994d6483441acad6640127 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad6915b950994d6483441acad6640127" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40777 } }
I20260923 17:28:49.881424 32533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:49.866678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38813 (local address 127.0.0.1:40777)
0923 17:28:49.867163 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:49.867179 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:49.867216 (+ 37us) server_negotiation.cc:407] Connection header received
0923 17:28:49.868037 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:49.868058 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:49.868412 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:49.868896 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:49.869746 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:49.870926 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:49.871870 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:49.872489 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:49.872697 (+ 208us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:49.875623 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:49.875628 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:49.875642 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:49.875669 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:49.878178 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:49.878792 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:49.878798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:49.878803 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:49.878921 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:49.879281 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:49.879284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:49.879286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:49.879807 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:49.880061 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:49.880351 (+ 290us) server_negotiation.cc:299] Negotiation successful
0923 17:28:49.880582 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":148,"threads_started":1}
I20260923 17:28:49.881760 32530 leader_election.cc:302] T 00000000000000000000000000000000 P ad6915b950994d6483441acad6640127 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad6915b950994d6483441acad6640127; no voters:
I20260923 17:28:49.882122 32530 leader_election.cc:288] T 00000000000000000000000000000000 P ad6915b950994d6483441acad6640127 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:28:49.882299 32535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad6915b950994d6483441acad6640127 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:49.882512 32535 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad6915b950994d6483441acad6640127 [term 1 LEADER]: Becoming Leader. State: Replica: ad6915b950994d6483441acad6640127, State: Running, Role: LEADER
I20260923 17:28:49.882828 32535 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad6915b950994d6483441acad6640127 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad6915b950994d6483441acad6640127" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40777 } }
I20260923 17:28:49.883443 32530 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad6915b950994d6483441acad6640127 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:28:49.884582 32537 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad6915b950994d6483441acad6640127 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad6915b950994d6483441acad6640127. Latest consensus state: current_term: 1 leader_uuid: "ad6915b950994d6483441acad6640127" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad6915b950994d6483441acad6640127" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40777 } } }
I20260923 17:28:49.884721 32537 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad6915b950994d6483441acad6640127 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:49.884984 12180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:28:49.884584 32536 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad6915b950994d6483441acad6640127 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad6915b950994d6483441acad6640127" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad6915b950994d6483441acad6640127" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40777 } } }
I20260923 17:28:49.885150 32536 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad6915b950994d6483441acad6640127 [sys.catalog]: This master's current role is: LEADER
W20260923 17:28:49.888631 32550 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ad6915b950994d6483441acad6640127: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:28:49.888808 32550 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:28:49.888975 32544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:28:49.889969 32544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:28:49.896538 32544 catalog_manager.cc:1347] Generated new cluster ID: 61738f0feda948d1b6c45362a7dd8777
I20260923 17:28:49.896606 32544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:28:49.900362 32533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:49.887252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49463 (local address 127.0.0.1:40777)
0923 17:28:49.887353 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:49.887358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:49.887729 (+ 371us) server_negotiation.cc:407] Connection header received
0923 17:28:49.887834 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:49.887837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:49.887885 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:49.887972 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:49.892288 (+ 4316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:49.893065 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:49.894328 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:49.894658 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:49.894783 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:49.896424 (+ 1641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:49.896430 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:49.896433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:49.896459 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:49.898394 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:49.898970 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:49.898974 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:49.898976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:49.899057 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:49.899391 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:49.899394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:49.899396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:49.899587 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:49.899705 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:49.900099 (+ 394us) server_negotiation.cc:299] Negotiation successful
0923 17:28:49.900199 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 17:28:49.906280 32544 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:28:49.907411 32544 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:28:49.924818 32544 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad6915b950994d6483441acad6640127: Generated new TSK 0
I20260923 17:28:49.925868 32544 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:28:49.934636 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 32462
W20260923 17:28:49.940629 32447 connection.cc:535] client connection to 127.0.0.1:40777 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40777): Connection reset by peer (error 104)
2026-09-23T17:28:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1440 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:28:49 dist-test-slave-ptl9 krb5kdc[32561](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:28:49 dist-test-slave-ptl9 krb5kdc[32561](info): set up 2 sockets
Sep 23 17:28:49 dist-test-slave-ptl9 krb5kdc[32561](info): commencing operation
krb5kdc: starting...
W20260923 17:28:50.989728 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:28:51 dist-test-slave-ptl9 krb5kdc[32561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:28:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:28:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:28:51.117092 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39955
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:51.239980 32574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:51.240345 32574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:51.240434 32574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:51.245007 32574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:28:51.245122 32574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:51.245165 32574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:51.245223 32574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:28:51.245309 32574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:28:51.251510 32574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39955
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:51.252745 32574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:51.253919 32574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:51.263063 32580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:51.263070 32579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:51.263095 32582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:51.263405 32574 server_base.cc:1034] running on GCE node
I20260923 17:28:51.263993 32574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:51.265100 32574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:51.266292 32574 hybrid_clock.cc:648] HybridClock initialized: now 1790184531266246 us; error 58 us; skew 500 ppm
Sep 23 17:28:51 dist-test-slave-ptl9 krb5kdc[32561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:51.269986 32574 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:28:51.270982 32574 webserver.cc:460] Webserver started at http://127.0.0.1:37677/ using document root <none> and password file <none>
I20260923 17:28:51.271569 32574 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:51.271636 32574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:51.271829 32574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:51.274317 32574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "ccc9b3e041d544dfbf996046996e1337"
format_stamp: "Formatted at 2026-09-23 17:28:51 on dist-test-slave-ptl9"
server_key: "5f5ee9c67282eef37c3a4898958445d4"
server_key_iv: "d560087840397bf9fcee197f257655ff"
server_key_version: "encryptionkey@0"
I20260923 17:28:51.274961 32574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "ccc9b3e041d544dfbf996046996e1337"
format_stamp: "Formatted at 2026-09-23 17:28:51 on dist-test-slave-ptl9"
server_key: "5f5ee9c67282eef37c3a4898958445d4"
server_key_iv: "d560087840397bf9fcee197f257655ff"
server_key_version: "encryptionkey@0"
I20260923 17:28:51.279059 32574 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:28:51.281778 32589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:51.283191 32574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:28:51.283368 32574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "ccc9b3e041d544dfbf996046996e1337"
format_stamp: "Formatted at 2026-09-23 17:28:51 on dist-test-slave-ptl9"
server_key: "5f5ee9c67282eef37c3a4898958445d4"
server_key_iv: "d560087840397bf9fcee197f257655ff"
server_key_version: "encryptionkey@0"
I20260923 17:28:51.283497 32574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:51.296629 32574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:51.300357 32574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:51.300592 32574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:51.308719 32574 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39955
I20260923 17:28:51.308769 32641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39955 every 8 connection(s)
I20260923 17:28:51.309872 32574 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:28:51.312813 32642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:51.314218 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 32574
I20260923 17:28:51.314399 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:28:51.314715 12180 external_mini_cluster.cc:1428] Setting key 7574c3ec58a8c4d9561062b2bfae6ffe
I20260923 17:28:51.320621 32642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccc9b3e041d544dfbf996046996e1337: Bootstrap starting.
I20260923 17:28:51.323760 32642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccc9b3e041d544dfbf996046996e1337: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:51.324554 32642 log.cc:824] T 00000000000000000000000000000000 P ccc9b3e041d544dfbf996046996e1337: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:51.324707 32645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:51.316265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58569 (local address 127.0.0.1:39955)
0923 17:28:51.316771 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:51.316790 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:51.316830 (+ 40us) server_negotiation.cc:407] Connection header received
0923 17:28:51.317614 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:51.317634 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:51.318009 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:51.318312 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:51.318736 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:51.319745 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:51.320999 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:51.321635 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:51.321775 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:51.322839 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:51.322845 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:51.322859 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:51.323275 (+ 416us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:51.323340 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:51.323365 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:51.323474 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:51.323691 (+ 217us) server_negotiation.cc:299] Negotiation successful
0923 17:28:51.323865 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":147,"threads_started":1}
I20260923 17:28:51.326864 32642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccc9b3e041d544dfbf996046996e1337: No bootstrap required, opened a new log
I20260923 17:28:51.328266 12180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:28:51.331830 32642 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccc9b3e041d544dfbf996046996e1337 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc9b3e041d544dfbf996046996e1337" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39955 } }
I20260923 17:28:51.332099 32642 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccc9b3e041d544dfbf996046996e1337 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:51.332155 32642 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccc9b3e041d544dfbf996046996e1337 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccc9b3e041d544dfbf996046996e1337, State: Initialized, Role: FOLLOWER
I20260923 17:28:51.332651 32642 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccc9b3e041d544dfbf996046996e1337 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc9b3e041d544dfbf996046996e1337" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39955 } }
I20260923 17:28:51.332832 32642 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccc9b3e041d544dfbf996046996e1337 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:28:51.332908 32642 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccc9b3e041d544dfbf996046996e1337 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:28:51.333012 32642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccc9b3e041d544dfbf996046996e1337 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:51.334335 32642 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccc9b3e041d544dfbf996046996e1337 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc9b3e041d544dfbf996046996e1337" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39955 } }
I20260923 17:28:51.334806 32642 leader_election.cc:302] T 00000000000000000000000000000000 P ccc9b3e041d544dfbf996046996e1337 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccc9b3e041d544dfbf996046996e1337; no voters:
I20260923 17:28:51.335263 32642 leader_election.cc:288] T 00000000000000000000000000000000 P ccc9b3e041d544dfbf996046996e1337 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:28:51.335470 32652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccc9b3e041d544dfbf996046996e1337 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:51.335656 32645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:51.330812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37423 (local address 127.0.0.1:39955)
0923 17:28:51.330957 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:51.330962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:51.331144 (+ 182us) server_negotiation.cc:407] Connection header received
0923 17:28:51.331288 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:51.331290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:51.331330 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:51.331391 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:51.331701 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:51.332400 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:51.333615 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:51.333879 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:51.333970 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:51.335044 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:51.335047 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:51.335050 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:51.335063 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:51.335094 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:51.335105 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:51.335178 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:51.335400 (+ 222us) server_negotiation.cc:299] Negotiation successful
0923 17:28:51.335444 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 17:28:51.336032 32652 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccc9b3e041d544dfbf996046996e1337 [term 1 LEADER]: Becoming Leader. State: Replica: ccc9b3e041d544dfbf996046996e1337, State: Running, Role: LEADER
I20260923 17:28:51.336429 32652 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccc9b3e041d544dfbf996046996e1337 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc9b3e041d544dfbf996046996e1337" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39955 } }
I20260923 17:28:51.336737 32642 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccc9b3e041d544dfbf996046996e1337 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:28:51.338163 32653 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccc9b3e041d544dfbf996046996e1337 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccc9b3e041d544dfbf996046996e1337" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc9b3e041d544dfbf996046996e1337" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39955 } } }
I20260923 17:28:51.338296 32653 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccc9b3e041d544dfbf996046996e1337 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:51.338645 32654 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccc9b3e041d544dfbf996046996e1337 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccc9b3e041d544dfbf996046996e1337. Latest consensus state: current_term: 1 leader_uuid: "ccc9b3e041d544dfbf996046996e1337" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc9b3e041d544dfbf996046996e1337" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39955 } } }
I20260923 17:28:51.338742 32654 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccc9b3e041d544dfbf996046996e1337 [sys.catalog]: This master's current role is: LEADER
W20260923 17:28:51.341746 32667 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ccc9b3e041d544dfbf996046996e1337: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:28:51.341840 32667 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:28:51.341989 32664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:28:51.343106 32664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:28:51.349660 32664 catalog_manager.cc:1347] Generated new cluster ID: edbb644817bb47e9b37c912e81443dae
I20260923 17:28:51.349769 32664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:28:51.369684 32664 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:28:51.371029 32664 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:28:51.378722 32664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccc9b3e041d544dfbf996046996e1337: Generated new TSK 0
I20260923 17:28:51.379563 32664 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:28:51.394387 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 32574
W20260923 17:28:51.400660 32559 connection.cc:535] client connection to 127.0.0.1:39955 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39955): Connection reset by peer (error 104)
2026-09-23T17:28:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1459 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:28:51 dist-test-slave-ptl9 krb5kdc[32673](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:28:51 dist-test-slave-ptl9 krb5kdc[32673](info): set up 2 sockets
Sep 23 17:28:51 dist-test-slave-ptl9 krb5kdc[32673](info): commencing operation
krb5kdc: starting...
W20260923 17:28:52.449766 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:28:52 dist-test-slave-ptl9 krb5kdc[32673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:28:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:28:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:28:52.581144 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40141
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:52.695158 32686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:52.695556 32686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:52.695672 32686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:52.699852 32686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:28:52.699967 32686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:52.700014 32686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:52.700052 32686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:28:52.700084 32686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:28:52.705647 32686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40141
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:52.706851 32686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:52.707932 32686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:52.716921 32692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:52.716939 32691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:52.716939 32694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:52.717177 32686 server_base.cc:1034] running on GCE node
I20260923 17:28:52.717846 32686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:52.719084 32686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:52.720310 32686 hybrid_clock.cc:648] HybridClock initialized: now 1790184532720290 us; error 31 us; skew 500 ppm
Sep 23 17:28:52 dist-test-slave-ptl9 krb5kdc[32673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:52.723295 32686 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:28:52.724221 32686 webserver.cc:460] Webserver started at http://127.0.0.1:33405/ using document root <none> and password file <none>
I20260923 17:28:52.724777 32686 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:52.724836 32686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:52.725004 32686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:52.727161 32686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "be6daacdaa254ac7a90575600fef551a"
format_stamp: "Formatted at 2026-09-23 17:28:52 on dist-test-slave-ptl9"
server_key: "903bf75f817a280ec952b5906e2354c9"
server_key_iv: "047c26dca5c71048111c95a811c716a6"
server_key_version: "encryptionkey@0"
I20260923 17:28:52.727707 32686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "be6daacdaa254ac7a90575600fef551a"
format_stamp: "Formatted at 2026-09-23 17:28:52 on dist-test-slave-ptl9"
server_key: "903bf75f817a280ec952b5906e2354c9"
server_key_iv: "047c26dca5c71048111c95a811c716a6"
server_key_version: "encryptionkey@0"
I20260923 17:28:52.731637 32686 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260923 17:28:52.734257 32701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:52.735781 32686 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:28:52.735945 32686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "be6daacdaa254ac7a90575600fef551a"
format_stamp: "Formatted at 2026-09-23 17:28:52 on dist-test-slave-ptl9"
server_key: "903bf75f817a280ec952b5906e2354c9"
server_key_iv: "047c26dca5c71048111c95a811c716a6"
server_key_version: "encryptionkey@0"
I20260923 17:28:52.736073 32686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:52.741135 32686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:52.750900 32686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:52.751137 32686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:52.758891 32686 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40141
I20260923 17:28:52.758909 32753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40141 every 8 connection(s)
I20260923 17:28:52.759984 32686 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:28:52.763062 32754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:52.768105 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 32686
I20260923 17:28:52.768276 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:28:52.768600 12180 external_mini_cluster.cc:1428] Setting key ba11dd75ab500224e3789fba44097ee3
I20260923 17:28:52.769824 32754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be6daacdaa254ac7a90575600fef551a: Bootstrap starting.
I20260923 17:28:52.772382 32754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be6daacdaa254ac7a90575600fef551a: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:52.773351 32754 log.cc:824] T 00000000000000000000000000000000 P be6daacdaa254ac7a90575600fef551a: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:52.774641 32757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:52.770228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39365 (local address 127.0.0.1:40141)
0923 17:28:52.770727 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:52.770743 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:52.770783 (+ 40us) server_negotiation.cc:407] Connection header received
0923 17:28:52.771488 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:52.771507 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:52.771901 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:52.772277 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:52.773158 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:52.773162 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:52.773174 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:52.773183 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:52.773245 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:52.773316 (+ 71us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:52.773444 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:52.773589 (+ 145us) server_negotiation.cc:299] Negotiation successful
0923 17:28:52.773761 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":144,"threads_started":1}
I20260923 17:28:52.775717 32754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be6daacdaa254ac7a90575600fef551a: No bootstrap required, opened a new log
I20260923 17:28:52.777618 12180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:28:52.780001 32754 raft_consensus.cc:348] T 00000000000000000000000000000000 P be6daacdaa254ac7a90575600fef551a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6daacdaa254ac7a90575600fef551a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40141 } }
I20260923 17:28:52.780319 32754 raft_consensus.cc:374] T 00000000000000000000000000000000 P be6daacdaa254ac7a90575600fef551a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:52.780387 32754 raft_consensus.cc:729] T 00000000000000000000000000000000 P be6daacdaa254ac7a90575600fef551a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be6daacdaa254ac7a90575600fef551a, State: Initialized, Role: FOLLOWER
I20260923 17:28:52.780907 32754 consensus_queue.cc:260] T 00000000000000000000000000000000 P be6daacdaa254ac7a90575600fef551a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6daacdaa254ac7a90575600fef551a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40141 } }
I20260923 17:28:52.781109 32754 raft_consensus.cc:388] T 00000000000000000000000000000000 P be6daacdaa254ac7a90575600fef551a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:28:52.781186 32754 raft_consensus.cc:482] T 00000000000000000000000000000000 P be6daacdaa254ac7a90575600fef551a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:28:52.781320 32754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be6daacdaa254ac7a90575600fef551a [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:52.781574 32757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:52.779674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56529 (local address 127.0.0.1:40141)
0923 17:28:52.779803 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:52.779808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:52.779991 (+ 183us) server_negotiation.cc:407] Connection header received
0923 17:28:52.780075 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:52.780077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:52.780116 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:52.780203 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:52.780987 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:52.780990 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:52.780993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:52.781005 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:52.781038 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:52.781054 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:52.781148 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:52.781380 (+ 232us) server_negotiation.cc:299] Negotiation successful
0923 17:28:52.781422 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 17:28:52.782447 32754 raft_consensus.cc:504] T 00000000000000000000000000000000 P be6daacdaa254ac7a90575600fef551a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6daacdaa254ac7a90575600fef551a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40141 } }
I20260923 17:28:52.782828 32754 leader_election.cc:302] T 00000000000000000000000000000000 P be6daacdaa254ac7a90575600fef551a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be6daacdaa254ac7a90575600fef551a; no voters:
I20260923 17:28:52.783114 32754 leader_election.cc:288] T 00000000000000000000000000000000 P be6daacdaa254ac7a90575600fef551a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:28:52.783223 32764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be6daacdaa254ac7a90575600fef551a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:52.783490 32764 raft_consensus.cc:686] T 00000000000000000000000000000000 P be6daacdaa254ac7a90575600fef551a [term 1 LEADER]: Becoming Leader. State: Replica: be6daacdaa254ac7a90575600fef551a, State: Running, Role: LEADER
I20260923 17:28:52.783809 32764 consensus_queue.cc:237] T 00000000000000000000000000000000 P be6daacdaa254ac7a90575600fef551a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6daacdaa254ac7a90575600fef551a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40141 } }
I20260923 17:28:52.784463 32754 sys_catalog.cc:564] T 00000000000000000000000000000000 P be6daacdaa254ac7a90575600fef551a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:28:52.785625 32765 sys_catalog.cc:455] T 00000000000000000000000000000000 P be6daacdaa254ac7a90575600fef551a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be6daacdaa254ac7a90575600fef551a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6daacdaa254ac7a90575600fef551a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40141 } } }
I20260923 17:28:52.785789 32765 sys_catalog.cc:458] T 00000000000000000000000000000000 P be6daacdaa254ac7a90575600fef551a [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:52.786186 305 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:28:52.787042 32766 sys_catalog.cc:455] T 00000000000000000000000000000000 P be6daacdaa254ac7a90575600fef551a [sys.catalog]: SysCatalogTable state changed. Reason: New leader be6daacdaa254ac7a90575600fef551a. Latest consensus state: current_term: 1 leader_uuid: "be6daacdaa254ac7a90575600fef551a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6daacdaa254ac7a90575600fef551a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40141 } } }
I20260923 17:28:52.787184 32766 sys_catalog.cc:458] T 00000000000000000000000000000000 P be6daacdaa254ac7a90575600fef551a [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:52.789604 305 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:28:52.795639 305 catalog_manager.cc:1347] Generated new cluster ID: 70e2878371774a35b48eafff342e0890
I20260923 17:28:52.795737 305 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:28:52.809108 305 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:28:52.809551 305 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:28:52.817154 305 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be6daacdaa254ac7a90575600fef551a: Generated new TSK 0
I20260923 17:28:52.817970 305 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:28:52.825706 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 32686
2026-09-23T17:28:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1430 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:28:52 dist-test-slave-ptl9 krb5kdc[317](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:28:52 dist-test-slave-ptl9 krb5kdc[317](info): set up 2 sockets
Sep 23 17:28:52 dist-test-slave-ptl9 krb5kdc[317](info): commencing operation
krb5kdc: starting...
W20260923 17:28:53.874046 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:28:53 dist-test-slave-ptl9 krb5kdc[317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:28:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:28:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:28:54.004493 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38925
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:54.120675 330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:54.121013 330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:54.121066 330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:54.125562 330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:28:54.125662 330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:54.125684 330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:54.125702 330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:28:54.125752 330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:28:54.132484 330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:54.134119 330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:54.135314 330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:54.144194 336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:54.144167 335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:54.144167 338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:54.144563 330 server_base.cc:1034] running on GCE node
I20260923 17:28:54.145319 330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:54.146523 330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:54.147704 330 hybrid_clock.cc:648] HybridClock initialized: now 1790184534147672 us; error 55 us; skew 500 ppm
Sep 23 17:28:54 dist-test-slave-ptl9 krb5kdc[317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:54.151348 330 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:28:54.152320 330 webserver.cc:460] Webserver started at http://127.0.0.1:38121/ using document root <none> and password file <none>
I20260923 17:28:54.152930 330 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:54.153025 330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:54.153357 330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:54.155553 330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "5376757480ed49ab912c844d2363d5ce"
format_stamp: "Formatted at 2026-09-23 17:28:54 on dist-test-slave-ptl9"
server_key: "f1144da4110c208d30327c427d1b3e64"
server_key_iv: "c6f9d8c2681ba0df02fa420e26def5b0"
server_key_version: "encryptionkey@0"
I20260923 17:28:54.156162 330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "5376757480ed49ab912c844d2363d5ce"
format_stamp: "Formatted at 2026-09-23 17:28:54 on dist-test-slave-ptl9"
server_key: "f1144da4110c208d30327c427d1b3e64"
server_key_iv: "c6f9d8c2681ba0df02fa420e26def5b0"
server_key_version: "encryptionkey@0"
I20260923 17:28:54.160132 330 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260923 17:28:54.162844 345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:54.164137 330 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:28:54.164290 330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "5376757480ed49ab912c844d2363d5ce"
format_stamp: "Formatted at 2026-09-23 17:28:54 on dist-test-slave-ptl9"
server_key: "f1144da4110c208d30327c427d1b3e64"
server_key_iv: "c6f9d8c2681ba0df02fa420e26def5b0"
server_key_version: "encryptionkey@0"
I20260923 17:28:54.164433 330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:54.187107 330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:54.190316 330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:54.190544 330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:54.197865 330 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38925
I20260923 17:28:54.197889 397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38925 every 8 connection(s)
I20260923 17:28:54.198848 330 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:28:54.201870 398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:54.201877 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 330
I20260923 17:28:54.202045 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:28:54.202361 12180 external_mini_cluster.cc:1428] Setting key db3e678e3b260aa71a1856685731144e
I20260923 17:28:54.210265 398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5376757480ed49ab912c844d2363d5ce: Bootstrap starting.
Sep 23 17:28:54 dist-test-slave-ptl9 krb5kdc[317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 17:28:54.213317 398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5376757480ed49ab912c844d2363d5ce: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:54.214428 398 log.cc:824] T 00000000000000000000000000000000 P 5376757480ed49ab912c844d2363d5ce: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:54.217063 398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5376757480ed49ab912c844d2363d5ce: No bootstrap required, opened a new log
I20260923 17:28:54.218772 401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:54.203835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35347 (local address 127.0.0.1:38925)
0923 17:28:54.204333 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:54.204354 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:54.204403 (+ 49us) server_negotiation.cc:407] Connection header received
0923 17:28:54.205168 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:54.205193 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:54.205715 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:54.206132 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:54.206768 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:54.208438 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:54.209382 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:54.210128 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:54.210324 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:54.213086 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:54.213096 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:54.213119 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:54.213162 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:54.215231 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:54.215710 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:54.215717 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:54.215724 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:54.215875 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:54.216159 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:54.216164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:54.216167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:54.216719 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:54.216924 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:54.217190 (+ 266us) server_negotiation.cc:299] Negotiation successful
0923 17:28:54.217753 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":136,"threads_started":1}
I20260923 17:28:54.221948 12180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:28:54.222168 398 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5376757480ed49ab912c844d2363d5ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5376757480ed49ab912c844d2363d5ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38925 } }
I20260923 17:28:54.222456 398 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5376757480ed49ab912c844d2363d5ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:54.222540 398 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5376757480ed49ab912c844d2363d5ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5376757480ed49ab912c844d2363d5ce, State: Initialized, Role: FOLLOWER
I20260923 17:28:54.223059 398 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5376757480ed49ab912c844d2363d5ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5376757480ed49ab912c844d2363d5ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38925 } }
I20260923 17:28:54.223235 398 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5376757480ed49ab912c844d2363d5ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:28:54.223325 398 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5376757480ed49ab912c844d2363d5ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:28:54.223448 398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5376757480ed49ab912c844d2363d5ce [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:54.224871 398 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5376757480ed49ab912c844d2363d5ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5376757480ed49ab912c844d2363d5ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38925 } }
I20260923 17:28:54.225317 398 leader_election.cc:302] T 00000000000000000000000000000000 P 5376757480ed49ab912c844d2363d5ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5376757480ed49ab912c844d2363d5ce; no voters:
I20260923 17:28:54.225704 398 leader_election.cc:288] T 00000000000000000000000000000000 P 5376757480ed49ab912c844d2363d5ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:28:54.225804 408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5376757480ed49ab912c844d2363d5ce [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:54.226058 408 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5376757480ed49ab912c844d2363d5ce [term 1 LEADER]: Becoming Leader. State: Replica: 5376757480ed49ab912c844d2363d5ce, State: Running, Role: LEADER
I20260923 17:28:54.226377 408 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5376757480ed49ab912c844d2363d5ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5376757480ed49ab912c844d2363d5ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38925 } }
I20260923 17:28:54.227115 398 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5376757480ed49ab912c844d2363d5ce [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:28:54.230414 409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5376757480ed49ab912c844d2363d5ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5376757480ed49ab912c844d2363d5ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5376757480ed49ab912c844d2363d5ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38925 } } }
I20260923 17:28:54.230569 409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5376757480ed49ab912c844d2363d5ce [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:54.231045 410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5376757480ed49ab912c844d2363d5ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5376757480ed49ab912c844d2363d5ce. Latest consensus state: current_term: 1 leader_uuid: "5376757480ed49ab912c844d2363d5ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5376757480ed49ab912c844d2363d5ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38925 } } }
I20260923 17:28:54.231186 410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5376757480ed49ab912c844d2363d5ce [sys.catalog]: This master's current role is: LEADER
W20260923 17:28:54.233163 422 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5376757480ed49ab912c844d2363d5ce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:28:54.233299 422 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:28:54.233398 424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:28:54.233769 401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:54.224176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51845 (local address 127.0.0.1:38925)
0923 17:28:54.224304 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:54.224309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:54.224603 (+ 294us) server_negotiation.cc:407] Connection header received
0923 17:28:54.224683 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:54.224689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:54.224754 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:54.224852 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:54.225608 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:54.226476 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:54.227641 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:54.227951 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:54.228060 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:54.229892 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:54.229896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:54.229899 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:54.229924 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:54.231747 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:54.232177 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:54.232180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:54.232183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:54.232296 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:54.232572 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:54.232576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:54.232578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:54.232837 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:54.232982 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:54.233306 (+ 324us) server_negotiation.cc:299] Negotiation successful
0923 17:28:54.233450 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 17:28:54.234421 424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:28:54.240564 424 catalog_manager.cc:1347] Generated new cluster ID: 341b6a5ca9694d0382a160eee2f2889a
I20260923 17:28:54.240649 424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:28:54.251554 424 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:28:54.252570 424 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:28:54.258505 424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5376757480ed49ab912c844d2363d5ce: Generated new TSK 0
I20260923 17:28:54.259316 424 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:28:54.265551 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 330
2026-09-23T17:28:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1439 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:28:54 dist-test-slave-ptl9 krb5kdc[429](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:28:54 dist-test-slave-ptl9 krb5kdc[429](info): set up 2 sockets
Sep 23 17:28:54 dist-test-slave-ptl9 krb5kdc[429](info): commencing operation
krb5kdc: starting...
W20260923 17:28:55.309671 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:28:55 dist-test-slave-ptl9 krb5kdc[429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:28:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:28:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:28:55.431945 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42919
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:55.544229 442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:55.544564 442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:55.544679 442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:55.548799 442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:28:55.548897 442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:55.548943 442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:55.549000 442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:28:55.549039 442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:28:55.554517 442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:55.555612 442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:55.556641 442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:55.565353 448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:55.565358 450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:55.565353 447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:55.565559 442 server_base.cc:1034] running on GCE node
I20260923 17:28:55.566337 442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:55.567559 442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:55.568768 442 hybrid_clock.cc:648] HybridClock initialized: now 1790184535568757 us; error 28 us; skew 500 ppm
Sep 23 17:28:55 dist-test-slave-ptl9 krb5kdc[429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:55.571749 442 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:28:55.572680 442 webserver.cc:460] Webserver started at http://127.0.0.1:39795/ using document root <none> and password file <none>
I20260923 17:28:55.573405 442 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:55.573508 442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:55.573761 442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:55.575959 442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "7802e52d473d4a188c8ef87669552777"
format_stamp: "Formatted at 2026-09-23 17:28:55 on dist-test-slave-ptl9"
server_key: "f7f8f40f5cea3c6605555b70430d2ffd"
server_key_iv: "b842627a137fde54a1819d71db9f94fd"
server_key_version: "encryptionkey@0"
I20260923 17:28:55.576573 442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "7802e52d473d4a188c8ef87669552777"
format_stamp: "Formatted at 2026-09-23 17:28:55 on dist-test-slave-ptl9"
server_key: "f7f8f40f5cea3c6605555b70430d2ffd"
server_key_iv: "b842627a137fde54a1819d71db9f94fd"
server_key_version: "encryptionkey@0"
I20260923 17:28:55.580519 442 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 17:28:55.583015 457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:55.584283 442 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:28:55.584542 442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "7802e52d473d4a188c8ef87669552777"
format_stamp: "Formatted at 2026-09-23 17:28:55 on dist-test-slave-ptl9"
server_key: "f7f8f40f5cea3c6605555b70430d2ffd"
server_key_iv: "b842627a137fde54a1819d71db9f94fd"
server_key_version: "encryptionkey@0"
I20260923 17:28:55.584668 442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:55.601008 442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:55.604455 442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:55.604665 442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:55.611653 442 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42919
I20260923 17:28:55.611707 509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42919 every 8 connection(s)
I20260923 17:28:55.612636 442 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:28:55.615515 510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:55.618446 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 442
I20260923 17:28:55.618614 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:28:55.618928 12180 external_mini_cluster.cc:1428] Setting key ddd2de2576c0164c2f7f715a692705d7
I20260923 17:28:55.621831 510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7802e52d473d4a188c8ef87669552777: Bootstrap starting.
I20260923 17:28:55.624558 510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7802e52d473d4a188c8ef87669552777: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:55.625365 510 log.cc:824] T 00000000000000000000000000000000 P 7802e52d473d4a188c8ef87669552777: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:55.627483 510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7802e52d473d4a188c8ef87669552777: No bootstrap required, opened a new log
I20260923 17:28:55.629026 513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:55.620378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45805 (local address 127.0.0.1:42919)
0923 17:28:55.620827 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:55.620846 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:55.620888 (+ 42us) server_negotiation.cc:407] Connection header received
0923 17:28:55.621713 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:55.621736 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:55.622160 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:55.622521 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:55.622923 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.624431 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.625498 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.626070 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.626271 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:55.627081 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:55.627087 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:55.627103 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:55.627601 (+ 498us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:55.627665 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:55.627694 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:55.627793 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:55.627928 (+ 135us) server_negotiation.cc:299] Negotiation successful
0923 17:28:55.628131 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":111,"threads_started":1}
I20260923 17:28:55.631219 510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7802e52d473d4a188c8ef87669552777 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7802e52d473d4a188c8ef87669552777" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42919 } }
I20260923 17:28:55.631475 510 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7802e52d473d4a188c8ef87669552777 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:55.631563 510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7802e52d473d4a188c8ef87669552777 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7802e52d473d4a188c8ef87669552777, State: Initialized, Role: FOLLOWER
I20260923 17:28:55.632050 510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7802e52d473d4a188c8ef87669552777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7802e52d473d4a188c8ef87669552777" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42919 } }
I20260923 17:28:55.632210 510 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7802e52d473d4a188c8ef87669552777 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:28:55.632302 510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7802e52d473d4a188c8ef87669552777 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:28:55.632401 510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7802e52d473d4a188c8ef87669552777 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:55.632429 12180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:28:55.633424 510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7802e52d473d4a188c8ef87669552777 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7802e52d473d4a188c8ef87669552777" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42919 } }
I20260923 17:28:55.633821 510 leader_election.cc:302] T 00000000000000000000000000000000 P 7802e52d473d4a188c8ef87669552777 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7802e52d473d4a188c8ef87669552777; no voters:
I20260923 17:28:55.634176 510 leader_election.cc:288] T 00000000000000000000000000000000 P 7802e52d473d4a188c8ef87669552777 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:28:55.634240 519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7802e52d473d4a188c8ef87669552777 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:55.634481 519 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7802e52d473d4a188c8ef87669552777 [term 1 LEADER]: Becoming Leader. State: Replica: 7802e52d473d4a188c8ef87669552777, State: Running, Role: LEADER
I20260923 17:28:55.634838 519 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7802e52d473d4a188c8ef87669552777 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7802e52d473d4a188c8ef87669552777" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42919 } }
I20260923 17:28:55.635547 510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7802e52d473d4a188c8ef87669552777 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:28:55.636158 519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7802e52d473d4a188c8ef87669552777 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7802e52d473d4a188c8ef87669552777. Latest consensus state: current_term: 1 leader_uuid: "7802e52d473d4a188c8ef87669552777" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7802e52d473d4a188c8ef87669552777" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42919 } } }
I20260923 17:28:55.636955 519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7802e52d473d4a188c8ef87669552777 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:55.636839 520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7802e52d473d4a188c8ef87669552777 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7802e52d473d4a188c8ef87669552777" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7802e52d473d4a188c8ef87669552777" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42919 } } }
I20260923 17:28:55.637297 520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7802e52d473d4a188c8ef87669552777 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:55.639473 513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:55.634896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49555 (local address 127.0.0.1:42919)
0923 17:28:55.635005 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:55.635010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:55.635118 (+ 108us) server_negotiation.cc:407] Connection header received
0923 17:28:55.635208 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:55.635211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:55.635262 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:55.635351 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:55.635619 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.636310 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.637477 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.637869 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.638004 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:55.639041 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:55.639044 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:55.639046 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:55.639057 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:55.639081 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:55.639092 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:55.639174 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:55.639265 (+ 91us) server_negotiation.cc:299] Negotiation successful
0923 17:28:55.639339 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260923 17:28:55.640915 536 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7802e52d473d4a188c8ef87669552777: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:28:55.640980 536 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:28:55.641059 529 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:28:55.641848 529 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:28:55.647720 529 catalog_manager.cc:1347] Generated new cluster ID: 061b4a116af944daa875114c0c19ebbb
I20260923 17:28:55.647797 529 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:28:55.663674 529 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:28:55.664942 529 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:28:55.676038 529 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7802e52d473d4a188c8ef87669552777: Generated new TSK 0
I20260923 17:28:55.676813 529 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:28:55.680883 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 442
2026-09-23T17:28:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1415 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:28:55 dist-test-slave-ptl9 krb5kdc[541](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:28:55 dist-test-slave-ptl9 krb5kdc[541](info): set up 2 sockets
Sep 23 17:28:55 dist-test-slave-ptl9 krb5kdc[541](info): commencing operation
krb5kdc: starting...
W20260923 17:28:56.725735 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:28:56 dist-test-slave-ptl9 krb5kdc[541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:28:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:28:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:28:56.845137 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39639
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:56.956308 554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:56.956631 554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:56.956702 554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:56.960752 554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:28:56.960843 554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:56.960880 554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:56.960939 554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:28:56.960978 554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:28:56.966419 554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:56.967561 554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:56.968605 554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:56.977357 562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:56.977377 559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:56.977437 554 server_base.cc:1034] running on GCE node
W20260923 17:28:56.977353 560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:56.978139 554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:56.979275 554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:56.980441 554 hybrid_clock.cc:648] HybridClock initialized: now 1790184536980430 us; error 24 us; skew 500 ppm
Sep 23 17:28:56 dist-test-slave-ptl9 krb5kdc[541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:56.983247 554 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:28:56.984110 554 webserver.cc:460] Webserver started at http://127.0.0.1:42623/ using document root <none> and password file <none>
I20260923 17:28:56.984620 554 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:56.984676 554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:56.984834 554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:56.986851 554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "973d352db8694fe4af47cff2f07140f1"
format_stamp: "Formatted at 2026-09-23 17:28:56 on dist-test-slave-ptl9"
server_key: "b1aa8e667b57b528a00999a67e530596"
server_key_iv: "aef06108b43816dd17d5f6f408a0b638"
server_key_version: "encryptionkey@0"
I20260923 17:28:56.987363 554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "973d352db8694fe4af47cff2f07140f1"
format_stamp: "Formatted at 2026-09-23 17:28:56 on dist-test-slave-ptl9"
server_key: "b1aa8e667b57b528a00999a67e530596"
server_key_iv: "aef06108b43816dd17d5f6f408a0b638"
server_key_version: "encryptionkey@0"
I20260923 17:28:56.991017 554 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260923 17:28:56.993445 569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:56.994619 554 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 17:28:56.994758 554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "973d352db8694fe4af47cff2f07140f1"
format_stamp: "Formatted at 2026-09-23 17:28:56 on dist-test-slave-ptl9"
server_key: "b1aa8e667b57b528a00999a67e530596"
server_key_iv: "aef06108b43816dd17d5f6f408a0b638"
server_key_version: "encryptionkey@0"
I20260923 17:28:56.994874 554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:56.999473 554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:57.002759 554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:57.002967 554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:57.010049 554 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39639
I20260923 17:28:57.010061 621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39639 every 8 connection(s)
I20260923 17:28:57.011009 554 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:28:57.013983 622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:57.019562 622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 973d352db8694fe4af47cff2f07140f1: Bootstrap starting.
I20260923 17:28:57.021116 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 554
I20260923 17:28:57.021282 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:28:57.021600 12180 external_mini_cluster.cc:1428] Setting key 9b80a44c517d9f028a23b38c54792fbc
I20260923 17:28:57.022579 622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 973d352db8694fe4af47cff2f07140f1: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:57.023414 622 log.cc:824] T 00000000000000000000000000000000 P 973d352db8694fe4af47cff2f07140f1: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:57.025766 622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 973d352db8694fe4af47cff2f07140f1: No bootstrap required, opened a new log
I20260923 17:28:57.026878 625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:57.022771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58741 (local address 127.0.0.1:39639)
0923 17:28:57.023202 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:57.023218 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:57.023254 (+ 36us) server_negotiation.cc:407] Connection header received
0923 17:28:57.024061 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:57.024087 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:57.024424 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:57.024709 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:57.025559 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:57.025564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:57.025575 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:57.025585 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:57.025643 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:57.025663 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:57.025750 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:57.025995 (+ 245us) server_negotiation.cc:299] Negotiation successful
0923 17:28:57.026165 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":129,"threads_started":1}
I20260923 17:28:57.029783 12180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:28:57.030047 622 raft_consensus.cc:348] T 00000000000000000000000000000000 P 973d352db8694fe4af47cff2f07140f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "973d352db8694fe4af47cff2f07140f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39639 } }
I20260923 17:28:57.030333 622 raft_consensus.cc:374] T 00000000000000000000000000000000 P 973d352db8694fe4af47cff2f07140f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:57.030393 622 raft_consensus.cc:729] T 00000000000000000000000000000000 P 973d352db8694fe4af47cff2f07140f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 973d352db8694fe4af47cff2f07140f1, State: Initialized, Role: FOLLOWER
I20260923 17:28:57.030869 622 consensus_queue.cc:260] T 00000000000000000000000000000000 P 973d352db8694fe4af47cff2f07140f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "973d352db8694fe4af47cff2f07140f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39639 } }
I20260923 17:28:57.031054 622 raft_consensus.cc:388] T 00000000000000000000000000000000 P 973d352db8694fe4af47cff2f07140f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:28:57.031136 622 raft_consensus.cc:482] T 00000000000000000000000000000000 P 973d352db8694fe4af47cff2f07140f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:28:57.031253 622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 973d352db8694fe4af47cff2f07140f1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:57.032522 622 raft_consensus.cc:504] T 00000000000000000000000000000000 P 973d352db8694fe4af47cff2f07140f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "973d352db8694fe4af47cff2f07140f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39639 } }
I20260923 17:28:57.032927 622 leader_election.cc:302] T 00000000000000000000000000000000 P 973d352db8694fe4af47cff2f07140f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 973d352db8694fe4af47cff2f07140f1; no voters:
I20260923 17:28:57.033334 622 leader_election.cc:288] T 00000000000000000000000000000000 P 973d352db8694fe4af47cff2f07140f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:28:57.033411 632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 973d352db8694fe4af47cff2f07140f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:57.033653 632 raft_consensus.cc:686] T 00000000000000000000000000000000 P 973d352db8694fe4af47cff2f07140f1 [term 1 LEADER]: Becoming Leader. State: Replica: 973d352db8694fe4af47cff2f07140f1, State: Running, Role: LEADER
I20260923 17:28:57.033838 625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:57.032067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49759 (local address 127.0.0.1:39639)
0923 17:28:57.032211 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:57.032216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:57.032376 (+ 160us) server_negotiation.cc:407] Connection header received
0923 17:28:57.032469 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:57.032475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:57.032515 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:57.032603 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:57.033411 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:57.033415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:57.033420 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:57.033435 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:57.033463 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:57.033476 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:57.033565 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:57.033659 (+ 94us) server_negotiation.cc:299] Negotiation successful
0923 17:28:57.033699 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 17:28:57.034751 622 sys_catalog.cc:564] T 00000000000000000000000000000000 P 973d352db8694fe4af47cff2f07140f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:28:57.034754 632 consensus_queue.cc:237] T 00000000000000000000000000000000 P 973d352db8694fe4af47cff2f07140f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "973d352db8694fe4af47cff2f07140f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39639 } }
I20260923 17:28:57.036640 634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 973d352db8694fe4af47cff2f07140f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 973d352db8694fe4af47cff2f07140f1. Latest consensus state: current_term: 1 leader_uuid: "973d352db8694fe4af47cff2f07140f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "973d352db8694fe4af47cff2f07140f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39639 } } }
I20260923 17:28:57.036675 633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 973d352db8694fe4af47cff2f07140f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "973d352db8694fe4af47cff2f07140f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "973d352db8694fe4af47cff2f07140f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39639 } } }
I20260923 17:28:57.036808 634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 973d352db8694fe4af47cff2f07140f1 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:57.036816 633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 973d352db8694fe4af47cff2f07140f1 [sys.catalog]: This master's current role is: LEADER
W20260923 17:28:57.040109 646 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 973d352db8694fe4af47cff2f07140f1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:28:57.040213 646 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:28:57.040324 647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:28:57.041049 647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:28:57.046351 647 catalog_manager.cc:1347] Generated new cluster ID: 8bd62f2644dd46a9bf1127e4fa81c8e3
I20260923 17:28:57.046428 647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:28:57.071676 647 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:28:57.072032 647 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:28:57.079475 647 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 973d352db8694fe4af47cff2f07140f1: Generated new TSK 0
I20260923 17:28:57.080318 647 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:28:57.087563 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 554
2026-09-23T17:28:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:28:57 dist-test-slave-ptl9 krb5kdc[653](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:28:57 dist-test-slave-ptl9 krb5kdc[653](info): set up 2 sockets
Sep 23 17:28:57 dist-test-slave-ptl9 krb5kdc[653](info): commencing operation
krb5kdc: starting...
W20260923 17:28:58.133607 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:28:58 dist-test-slave-ptl9 krb5kdc[653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:28:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:28:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:28:58.258154 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32997
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:58.378008 666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:58.378293 666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:58.378382 666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:58.382529 666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:28:58.382596 666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:58.382617 666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:58.382634 666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:28:58.382676 666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:28:58.388195 666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:58.389328 666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:58.390861 666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:58.399624 671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:58.399624 672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:58.399760 666 server_base.cc:1034] running on GCE node
W20260923 17:28:58.399619 674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:58.400648 666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:58.401904 666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:58.403113 666 hybrid_clock.cc:648] HybridClock initialized: now 1790184538403095 us; error 31 us; skew 500 ppm
Sep 23 17:28:58 dist-test-slave-ptl9 krb5kdc[653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:58.410095 666 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:28:58.411106 666 webserver.cc:460] Webserver started at http://127.0.0.1:36577/ using document root <none> and password file <none>
I20260923 17:28:58.411720 666 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:58.411823 666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:58.412068 666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:58.414417 666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "fb305459393f429aace075a2797a2c17"
format_stamp: "Formatted at 2026-09-23 17:28:58 on dist-test-slave-ptl9"
server_key: "0876e64c9471cc3cc620dc9489b07465"
server_key_iv: "1f35bc68a3862a01012b75f72ab02192"
server_key_version: "encryptionkey@0"
I20260923 17:28:58.415014 666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "fb305459393f429aace075a2797a2c17"
format_stamp: "Formatted at 2026-09-23 17:28:58 on dist-test-slave-ptl9"
server_key: "0876e64c9471cc3cc620dc9489b07465"
server_key_iv: "1f35bc68a3862a01012b75f72ab02192"
server_key_version: "encryptionkey@0"
I20260923 17:28:58.418964 666 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:28:58.421554 681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:58.422818 666 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:28:58.422986 666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "fb305459393f429aace075a2797a2c17"
format_stamp: "Formatted at 2026-09-23 17:28:58 on dist-test-slave-ptl9"
server_key: "0876e64c9471cc3cc620dc9489b07465"
server_key_iv: "1f35bc68a3862a01012b75f72ab02192"
server_key_version: "encryptionkey@0"
I20260923 17:28:58.423110 666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:58.443902 666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:58.447101 666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:58.447324 666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:58.455168 666 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32997
I20260923 17:28:58.455188 733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32997 every 8 connection(s)
I20260923 17:28:58.456177 666 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:28:58.459117 734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:58.464449 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 666
I20260923 17:28:58.464603 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:28:58.464931 12180 external_mini_cluster.cc:1428] Setting key 225ccc66be5be616ec0af6bea39a5e4f
I20260923 17:28:58.467067 734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb305459393f429aace075a2797a2c17: Bootstrap starting.
I20260923 17:28:58.470080 734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb305459393f429aace075a2797a2c17: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:58.470836 734 log.cc:824] T 00000000000000000000000000000000 P fb305459393f429aace075a2797a2c17: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:58.473323 734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb305459393f429aace075a2797a2c17: No bootstrap required, opened a new log
Sep 23 17:28:58 dist-test-slave-ptl9 krb5kdc[653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 17:28:58.477531 734 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb305459393f429aace075a2797a2c17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb305459393f429aace075a2797a2c17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32997 } }
I20260923 17:28:58.477787 734 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb305459393f429aace075a2797a2c17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:58.477857 734 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb305459393f429aace075a2797a2c17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb305459393f429aace075a2797a2c17, State: Initialized, Role: FOLLOWER
I20260923 17:28:58.478329 734 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb305459393f429aace075a2797a2c17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb305459393f429aace075a2797a2c17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32997 } }
I20260923 17:28:58.478451 734 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb305459393f429aace075a2797a2c17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:28:58.478536 734 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb305459393f429aace075a2797a2c17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:28:58.478648 734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb305459393f429aace075a2797a2c17 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:58.479863 734 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb305459393f429aace075a2797a2c17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb305459393f429aace075a2797a2c17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32997 } }
I20260923 17:28:58.480229 734 leader_election.cc:302] T 00000000000000000000000000000000 P fb305459393f429aace075a2797a2c17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb305459393f429aace075a2797a2c17; no voters:
I20260923 17:28:58.480562 734 leader_election.cc:288] T 00000000000000000000000000000000 P fb305459393f429aace075a2797a2c17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:28:58.480659 739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb305459393f429aace075a2797a2c17 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:58.480886 739 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb305459393f429aace075a2797a2c17 [term 1 LEADER]: Becoming Leader. State: Replica: fb305459393f429aace075a2797a2c17, State: Running, Role: LEADER
I20260923 17:28:58.481177 739 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb305459393f429aace075a2797a2c17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb305459393f429aace075a2797a2c17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32997 } }
I20260923 17:28:58.481930 734 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb305459393f429aace075a2797a2c17 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:28:58.482388 737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:58.466559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:60223 (local address 127.0.0.1:32997)
0923 17:28:58.467125 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:58.467146 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:58.467207 (+ 61us) server_negotiation.cc:407] Connection header received
0923 17:28:58.468153 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:58.468174 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:58.468573 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:58.468967 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:58.469845 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:58.471644 (+ 1799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:58.472903 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:58.473645 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:58.473836 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:58.476680 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:58.476685 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:58.476699 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:58.476727 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:58.478823 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:58.479388 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:58.479393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:58.479398 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:58.479503 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:58.479870 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:58.479872 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:58.479874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:58.480377 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:58.480612 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:58.481243 (+ 631us) server_negotiation.cc:299] Negotiation successful
0923 17:28:58.481527 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":157,"threads_started":1}
I20260923 17:28:58.482405 739 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb305459393f429aace075a2797a2c17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb305459393f429aace075a2797a2c17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb305459393f429aace075a2797a2c17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32997 } } }
I20260923 17:28:58.482535 739 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb305459393f429aace075a2797a2c17 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:58.482841 740 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb305459393f429aace075a2797a2c17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb305459393f429aace075a2797a2c17. Latest consensus state: current_term: 1 leader_uuid: "fb305459393f429aace075a2797a2c17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb305459393f429aace075a2797a2c17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32997 } } }
I20260923 17:28:58.483703 740 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb305459393f429aace075a2797a2c17 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:58.483973 750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:28:58.485746 12180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:28:58.487018 750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:28:58.493643 750 catalog_manager.cc:1347] Generated new cluster ID: 8cfc29fa1a5e43ae9f8d223178a90f42
I20260923 17:28:58.494737 750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:28:58.498548 737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:58.487904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:55321 (local address 127.0.0.1:32997)
0923 17:28:58.488007 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:58.488012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:58.489621 (+ 1609us) server_negotiation.cc:407] Connection header received
0923 17:28:58.489799 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:58.489803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:58.489859 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:58.489960 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:58.490760 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:58.491354 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:58.492593 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:58.492900 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:58.493030 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:58.494743 (+ 1713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:58.494750 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:58.494757 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:58.494790 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:58.496543 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:58.497129 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:58.497133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:58.497135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:58.497222 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:58.497595 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:58.497598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:58.497600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:58.497816 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:58.497963 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:58.498273 (+ 310us) server_negotiation.cc:299] Negotiation successful
0923 17:28:58.498377 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 17:28:58.503167 750 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:28:58.504083 750 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:28:58.508080 750 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb305459393f429aace075a2797a2c17: Generated new TSK 0
I20260923 17:28:58.508777 750 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:28:58.512908 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 666
W20260923 17:28:58.518842 651 connection.cc:535] client connection to 127.0.0.1:32997 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32997): Connection reset by peer (error 104)
2026-09-23T17:28:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1425 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:28:58 dist-test-slave-ptl9 krb5kdc[765](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:28:58 dist-test-slave-ptl9 krb5kdc[765](info): set up 2 sockets
Sep 23 17:28:58 dist-test-slave-ptl9 krb5kdc[765](info): commencing operation
krb5kdc: starting...
W20260923 17:28:59.561677 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:28:59 dist-test-slave-ptl9 krb5kdc[765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:28:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:28:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:28:59.687729 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38319
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:59.798059 778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:59.798388 778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:59.798480 778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:59.802517 778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:28:59.802618 778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:59.802659 778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:59.802696 778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:28:59.802731 778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:28:59.808040 778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:28:59.809137 778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:59.810184 778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:59.818904 783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:59.818889 786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:59.818889 784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:59.818966 778 server_base.cc:1034] running on GCE node
I20260923 17:28:59.819689 778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:59.820808 778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:59.822003 778 hybrid_clock.cc:648] HybridClock initialized: now 1790184539821992 us; error 29 us; skew 500 ppm
Sep 23 17:28:59 dist-test-slave-ptl9 krb5kdc[765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:59.824762 778 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:28:59.825763 778 webserver.cc:460] Webserver started at http://127.0.0.1:39745/ using document root <none> and password file <none>
I20260923 17:28:59.826344 778 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:59.826435 778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:59.826668 778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:59.828734 778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "1a3d0c7aebf7451d9f2659b3f46e9118"
format_stamp: "Formatted at 2026-09-23 17:28:59 on dist-test-slave-ptl9"
server_key: "10cd73b21b4e19ba40a47e6b7662c410"
server_key_iv: "650a9439134bc700bebe9e3b6e0db356"
server_key_version: "encryptionkey@0"
I20260923 17:28:59.829357 778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "1a3d0c7aebf7451d9f2659b3f46e9118"
format_stamp: "Formatted at 2026-09-23 17:28:59 on dist-test-slave-ptl9"
server_key: "10cd73b21b4e19ba40a47e6b7662c410"
server_key_iv: "650a9439134bc700bebe9e3b6e0db356"
server_key_version: "encryptionkey@0"
I20260923 17:28:59.833102 778 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:28:59.835697 793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:59.836931 778 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:28:59.837100 778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "1a3d0c7aebf7451d9f2659b3f46e9118"
format_stamp: "Formatted at 2026-09-23 17:28:59 on dist-test-slave-ptl9"
server_key: "10cd73b21b4e19ba40a47e6b7662c410"
server_key_iv: "650a9439134bc700bebe9e3b6e0db356"
server_key_version: "encryptionkey@0"
I20260923 17:28:59.837221 778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:59.850531 778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:59.853528 778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:59.853740 778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:59.861018 778 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38319
I20260923 17:28:59.861042 845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38319 every 8 connection(s)
I20260923 17:28:59.862068 778 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:28:59.863988 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 778
I20260923 17:28:59.864142 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:28:59.864449 12180 external_mini_cluster.cc:1428] Setting key 3ae75998316433906a8e54415c48ee3a
I20260923 17:28:59.865186 846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:59.871121 846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a3d0c7aebf7451d9f2659b3f46e9118: Bootstrap starting.
I20260923 17:28:59.873579 849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:59.865896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:60757 (local address 127.0.0.1:38319)
0923 17:28:59.866351 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:59.866365 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:59.866401 (+ 36us) server_negotiation.cc:407] Connection header received
0923 17:28:59.867051 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:59.867067 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:59.867399 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:59.867734 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:59.868141 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:59.869114 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:59.870059 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:59.870555 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:59.870736 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:59.871697 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:59.871702 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:59.871716 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:59.872112 (+ 396us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:59.872165 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:59.872187 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:59.872334 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:59.872527 (+ 193us) server_negotiation.cc:299] Negotiation successful
0923 17:28:59.872678 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":164,"threads_started":1}
I20260923 17:28:59.873664 846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a3d0c7aebf7451d9f2659b3f46e9118: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:59.874395 846 log.cc:824] T 00000000000000000000000000000000 P 1a3d0c7aebf7451d9f2659b3f46e9118: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:59.876534 846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a3d0c7aebf7451d9f2659b3f46e9118: No bootstrap required, opened a new log
I20260923 17:28:59.876567 12180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:28:59.880812 846 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a3d0c7aebf7451d9f2659b3f46e9118 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a3d0c7aebf7451d9f2659b3f46e9118" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38319 } }
I20260923 17:28:59.881089 846 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a3d0c7aebf7451d9f2659b3f46e9118 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:59.881194 846 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a3d0c7aebf7451d9f2659b3f46e9118 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a3d0c7aebf7451d9f2659b3f46e9118, State: Initialized, Role: FOLLOWER
I20260923 17:28:59.881728 846 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a3d0c7aebf7451d9f2659b3f46e9118 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a3d0c7aebf7451d9f2659b3f46e9118" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38319 } }
I20260923 17:28:59.881912 846 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a3d0c7aebf7451d9f2659b3f46e9118 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:28:59.882041 846 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a3d0c7aebf7451d9f2659b3f46e9118 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:28:59.882171 846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a3d0c7aebf7451d9f2659b3f46e9118 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:59.882975 849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:59.878835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:51743 (local address 127.0.0.1:38319)
0923 17:28:59.878977 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:59.878984 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:59.879175 (+ 191us) server_negotiation.cc:407] Connection header received
0923 17:28:59.879332 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:59.879335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:59.879383 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:59.879470 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:59.879740 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:59.880244 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:59.881069 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:59.881452 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:59.881560 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:59.882482 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:59.882485 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:59.882488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:59.882502 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:59.882532 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:59.882542 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:59.882646 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:59.882768 (+ 122us) server_negotiation.cc:299] Negotiation successful
0923 17:28:59.882805 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 17:28:59.883188 846 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a3d0c7aebf7451d9f2659b3f46e9118 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a3d0c7aebf7451d9f2659b3f46e9118" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38319 } }
I20260923 17:28:59.883541 846 leader_election.cc:302] T 00000000000000000000000000000000 P 1a3d0c7aebf7451d9f2659b3f46e9118 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a3d0c7aebf7451d9f2659b3f46e9118; no voters:
I20260923 17:28:59.883888 846 leader_election.cc:288] T 00000000000000000000000000000000 P 1a3d0c7aebf7451d9f2659b3f46e9118 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:28:59.883989 856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a3d0c7aebf7451d9f2659b3f46e9118 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:59.884195 856 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a3d0c7aebf7451d9f2659b3f46e9118 [term 1 LEADER]: Becoming Leader. State: Replica: 1a3d0c7aebf7451d9f2659b3f46e9118, State: Running, Role: LEADER
I20260923 17:28:59.884541 856 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a3d0c7aebf7451d9f2659b3f46e9118 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a3d0c7aebf7451d9f2659b3f46e9118" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38319 } }
I20260923 17:28:59.885317 846 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a3d0c7aebf7451d9f2659b3f46e9118 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:28:59.886303 858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a3d0c7aebf7451d9f2659b3f46e9118 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a3d0c7aebf7451d9f2659b3f46e9118" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a3d0c7aebf7451d9f2659b3f46e9118" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38319 } } }
I20260923 17:28:59.886446 858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a3d0c7aebf7451d9f2659b3f46e9118 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:59.886718 857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a3d0c7aebf7451d9f2659b3f46e9118 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a3d0c7aebf7451d9f2659b3f46e9118. Latest consensus state: current_term: 1 leader_uuid: "1a3d0c7aebf7451d9f2659b3f46e9118" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a3d0c7aebf7451d9f2659b3f46e9118" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38319 } } }
I20260923 17:28:59.886797 857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a3d0c7aebf7451d9f2659b3f46e9118 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:59.886825 865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:28:59.890048 865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:28:59.896073 865 catalog_manager.cc:1347] Generated new cluster ID: 305ab2151d4a4cb79628f1de5533f2cc
I20260923 17:28:59.896155 865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:28:59.916064 865 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:28:59.917057 865 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:28:59.928134 865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a3d0c7aebf7451d9f2659b3f46e9118: Generated new TSK 0
I20260923 17:28:59.928870 865 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:28:59.932957 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 778
W20260923 17:28:59.938930 763 connection.cc:535] client connection to 127.0.0.1:38319 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38319): Connection reset by peer (error 104)
2026-09-23T17:28:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1419 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:28:59 dist-test-slave-ptl9 krb5kdc[877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:28:59 dist-test-slave-ptl9 krb5kdc[877](info): set up 2 sockets
Sep 23 17:28:59 dist-test-slave-ptl9 krb5kdc[877](info): commencing operation
krb5kdc: starting...
W20260923 17:29:00.993716 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:29:01 dist-test-slave-ptl9 krb5kdc[877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:29:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:29:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:29:01.112903 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42961
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:29:01.224519 890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:01.224833 890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:01.224911 890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:29:01.228952 890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:29:01.229048 890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:01.229086 890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:01.229142 890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:29:01.229182 890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:29:01.234562 890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:29:01.235659 890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:01.236670 890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:01.245113 896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:01.245110 898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:01.245110 895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:01.245594 890 server_base.cc:1034] running on GCE node
I20260923 17:29:01.246070 890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:01.247068 890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:01.248260 890 hybrid_clock.cc:648] HybridClock initialized: now 1790184541248242 us; error 28 us; skew 500 ppm
Sep 23 17:29:01 dist-test-slave-ptl9 krb5kdc[877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184541, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:29:01.250968 890 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:29:01.251796 890 webserver.cc:460] Webserver started at http://127.0.0.1:36747/ using document root <none> and password file <none>
I20260923 17:29:01.252378 890 fs_manager.cc:359] Metadata directory not provided
I20260923 17:29:01.252470 890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:29:01.252697 890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:29:01.254737 890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "85a779dcfefe4f8c8bf08b6fb82df36d"
format_stamp: "Formatted at 2026-09-23 17:29:01 on dist-test-slave-ptl9"
server_key: "c274982713e85bb4d2cf40a3476edc75"
server_key_iv: "f626bc1bf84984b1e0f25285b513855d"
server_key_version: "encryptionkey@0"
I20260923 17:29:01.255300 890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "85a779dcfefe4f8c8bf08b6fb82df36d"
format_stamp: "Formatted at 2026-09-23 17:29:01 on dist-test-slave-ptl9"
server_key: "c274982713e85bb4d2cf40a3476edc75"
server_key_iv: "f626bc1bf84984b1e0f25285b513855d"
server_key_version: "encryptionkey@0"
I20260923 17:29:01.258929 890 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 17:29:01.261198 905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:01.262428 890 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:29:01.262566 890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "85a779dcfefe4f8c8bf08b6fb82df36d"
format_stamp: "Formatted at 2026-09-23 17:29:01 on dist-test-slave-ptl9"
server_key: "c274982713e85bb4d2cf40a3476edc75"
server_key_iv: "f626bc1bf84984b1e0f25285b513855d"
server_key_version: "encryptionkey@0"
I20260923 17:29:01.262679 890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:29:01.267184 890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:29:01.270438 890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:29:01.270633 890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:29:01.277658 890 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42961
I20260923 17:29:01.277679 957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42961 every 8 connection(s)
I20260923 17:29:01.278597 890 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:29:01.281524 958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:29:01.287096 958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85a779dcfefe4f8c8bf08b6fb82df36d: Bootstrap starting.
I20260923 17:29:01.288892 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 890
I20260923 17:29:01.289070 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:29:01.289453 12180 external_mini_cluster.cc:1428] Setting key e85eb20d39c2719ef8e56a896d44f65f
I20260923 17:29:01.289827 958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85a779dcfefe4f8c8bf08b6fb82df36d: Neither blocks nor log segments found. Creating new log.
I20260923 17:29:01.290741 958 log.cc:824] T 00000000000000000000000000000000 P 85a779dcfefe4f8c8bf08b6fb82df36d: Log is configured to *not* fsync() on all Append() calls
I20260923 17:29:01.293004 958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85a779dcfefe4f8c8bf08b6fb82df36d: No bootstrap required, opened a new log
I20260923 17:29:01.294986 961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:01.290645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:59391 (local address 127.0.0.1:42961)
0923 17:29:01.291154 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:01.291175 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:01.291218 (+ 43us) server_negotiation.cc:407] Connection header received
0923 17:29:01.292032 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:01.292057 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:01.292396 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:01.292760 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:01.293588 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:01.293594 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:01.293609 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:29:01.293621 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:01.293682 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:29:01.293709 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:01.293820 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:01.293911 (+ 91us) server_negotiation.cc:299] Negotiation successful
0923 17:29:01.294069 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":156,"threads_started":1}
I20260923 17:29:01.297473 958 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85a779dcfefe4f8c8bf08b6fb82df36d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a779dcfefe4f8c8bf08b6fb82df36d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42961 } }
I20260923 17:29:01.297734 958 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85a779dcfefe4f8c8bf08b6fb82df36d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:29:01.297780 958 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85a779dcfefe4f8c8bf08b6fb82df36d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a779dcfefe4f8c8bf08b6fb82df36d, State: Initialized, Role: FOLLOWER
I20260923 17:29:01.298112 12180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:29:01.298231 958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85a779dcfefe4f8c8bf08b6fb82df36d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a779dcfefe4f8c8bf08b6fb82df36d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42961 } }
I20260923 17:29:01.298372 958 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85a779dcfefe4f8c8bf08b6fb82df36d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:29:01.298429 958 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85a779dcfefe4f8c8bf08b6fb82df36d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:29:01.298519 958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85a779dcfefe4f8c8bf08b6fb82df36d [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:29:01.299777 958 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85a779dcfefe4f8c8bf08b6fb82df36d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a779dcfefe4f8c8bf08b6fb82df36d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42961 } }
I20260923 17:29:01.300153 958 leader_election.cc:302] T 00000000000000000000000000000000 P 85a779dcfefe4f8c8bf08b6fb82df36d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85a779dcfefe4f8c8bf08b6fb82df36d; no voters:
I20260923 17:29:01.300536 958 leader_election.cc:288] T 00000000000000000000000000000000 P 85a779dcfefe4f8c8bf08b6fb82df36d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:29:01.300928 968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85a779dcfefe4f8c8bf08b6fb82df36d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:29:01.301483 968 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85a779dcfefe4f8c8bf08b6fb82df36d [term 1 LEADER]: Becoming Leader. State: Replica: 85a779dcfefe4f8c8bf08b6fb82df36d, State: Running, Role: LEADER
I20260923 17:29:01.301826 968 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85a779dcfefe4f8c8bf08b6fb82df36d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a779dcfefe4f8c8bf08b6fb82df36d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42961 } }
I20260923 17:29:01.302035 958 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85a779dcfefe4f8c8bf08b6fb82df36d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:29:01.302322 961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:01.300417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:39735 (local address 127.0.0.1:42961)
0923 17:29:01.300550 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:01.300557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:01.300678 (+ 121us) server_negotiation.cc:407] Connection header received
0923 17:29:01.300817 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:01.300820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:01.300855 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:01.301215 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:01.301690 (+ 475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:01.301693 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:01.301695 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:29:01.301706 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:01.301732 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:29:01.301744 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:01.302141 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:01.302176 (+ 35us) server_negotiation.cc:299] Negotiation successful
0923 17:29:01.302214 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 17:29:01.303532 970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85a779dcfefe4f8c8bf08b6fb82df36d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85a779dcfefe4f8c8bf08b6fb82df36d. Latest consensus state: current_term: 1 leader_uuid: "85a779dcfefe4f8c8bf08b6fb82df36d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a779dcfefe4f8c8bf08b6fb82df36d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42961 } } }
I20260923 17:29:01.303678 970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85a779dcfefe4f8c8bf08b6fb82df36d [sys.catalog]: This master's current role is: LEADER
I20260923 17:29:01.304085 981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:29:01.304035 969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85a779dcfefe4f8c8bf08b6fb82df36d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85a779dcfefe4f8c8bf08b6fb82df36d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a779dcfefe4f8c8bf08b6fb82df36d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42961 } } }
I20260923 17:29:01.304541 969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85a779dcfefe4f8c8bf08b6fb82df36d [sys.catalog]: This master's current role is: LEADER
I20260923 17:29:01.307264 981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:29:01.313376 981 catalog_manager.cc:1347] Generated new cluster ID: 03e29c38af4547ae9352464c9fa2dedb
I20260923 17:29:01.313484 981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:29:01.336099 981 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:29:01.336436 981 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:29:01.340731 981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85a779dcfefe4f8c8bf08b6fb82df36d: Generated new TSK 0
I20260923 17:29:01.341481 981 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:29:01.352495 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 890
2026-09-23T17:29:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1419 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:29:01 dist-test-slave-ptl9 krb5kdc[989](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:29:01 dist-test-slave-ptl9 krb5kdc[989](info): set up 2 sockets
Sep 23 17:29:01 dist-test-slave-ptl9 krb5kdc[989](info): commencing operation
krb5kdc: starting...
W20260923 17:29:02.409723 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:29:02 dist-test-slave-ptl9 krb5kdc[989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:29:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:29:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:29:02.532531 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33537
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:29:02.643714 1002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:02.644017 1002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:02.644093 1002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:29:02.648065 1002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:29:02.648165 1002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:02.648204 1002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:02.648265 1002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:29:02.648305 1002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:29:02.653483 1002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33537
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:29:02.654541 1002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:02.655558 1002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:02.664019 1007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:02.664004 1008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:02.663995 1010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:02.664074 1002 server_base.cc:1034] running on GCE node
I20260923 17:29:02.664757 1002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:02.665939 1002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:02.667132 1002 hybrid_clock.cc:648] HybridClock initialized: now 1790184542667119 us; error 26 us; skew 500 ppm
Sep 23 17:29:02 dist-test-slave-ptl9 krb5kdc[989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184542, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:29:02.669992 1002 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:29:02.670907 1002 webserver.cc:460] Webserver started at http://127.0.0.1:46433/ using document root <none> and password file <none>
I20260923 17:29:02.671702 1002 fs_manager.cc:359] Metadata directory not provided
I20260923 17:29:02.671839 1002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:29:02.672039 1002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:29:02.674276 1002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "d2e178ec907c4353af49d4f658f32c9b"
format_stamp: "Formatted at 2026-09-23 17:29:02 on dist-test-slave-ptl9"
server_key: "c8f3ad22a408c373fc31196e6f5d9889"
server_key_iv: "9db25d5c3ac5d9d00e5f00a1da382bc5"
server_key_version: "encryptionkey@0"
I20260923 17:29:02.674803 1002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "d2e178ec907c4353af49d4f658f32c9b"
format_stamp: "Formatted at 2026-09-23 17:29:02 on dist-test-slave-ptl9"
server_key: "c8f3ad22a408c373fc31196e6f5d9889"
server_key_iv: "9db25d5c3ac5d9d00e5f00a1da382bc5"
server_key_version: "encryptionkey@0"
I20260923 17:29:02.678617 1002 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:29:02.680971 1017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:02.682353 1002 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:29:02.682466 1002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "d2e178ec907c4353af49d4f658f32c9b"
format_stamp: "Formatted at 2026-09-23 17:29:02 on dist-test-slave-ptl9"
server_key: "c8f3ad22a408c373fc31196e6f5d9889"
server_key_iv: "9db25d5c3ac5d9d00e5f00a1da382bc5"
server_key_version: "encryptionkey@0"
I20260923 17:29:02.682549 1002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:29:02.695914 1002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:29:02.698746 1002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:29:02.698904 1002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:29:02.706157 1002 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33537
I20260923 17:29:02.706167 1069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33537 every 8 connection(s)
I20260923 17:29:02.707134 1002 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:29:02.708482 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 1002
I20260923 17:29:02.708637 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:29:02.708936 12180 external_mini_cluster.cc:1428] Setting key e2d987088e22e959d61b33444577b2a3
I20260923 17:29:02.710420 1070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:29:02.716109 1070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2e178ec907c4353af49d4f658f32c9b: Bootstrap starting.
Sep 23 17:29:02 dist-test-slave-ptl9 krb5kdc[989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 17:29:02.718770 1070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2e178ec907c4353af49d4f658f32c9b: Neither blocks nor log segments found. Creating new log.
I20260923 17:29:02.719650 1070 log.cc:824] T 00000000000000000000000000000000 P d2e178ec907c4353af49d4f658f32c9b: Log is configured to *not* fsync() on all Append() calls
I20260923 17:29:02.721994 1070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2e178ec907c4353af49d4f658f32c9b: No bootstrap required, opened a new log
I20260923 17:29:02.723302 1073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:02.710292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:49003 (local address 127.0.0.1:33537)
0923 17:29:02.710784 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:02.710799 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:02.710837 (+ 38us) server_negotiation.cc:407] Connection header received
0923 17:29:02.711513 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:02.711529 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:02.711892 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:02.712234 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:02.713058 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.714173 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.715042 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.715520 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.715686 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:02.718472 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:02.718477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:02.718492 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:02.718518 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:02.720346 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:02.720924 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:02.720929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:02.720934 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:02.721066 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:02.721428 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:02.721431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:02.721432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:02.721819 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:02.721980 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:02.722328 (+ 348us) server_negotiation.cc:299] Negotiation successful
0923 17:29:02.722578 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":153,"threads_started":1}
I20260923 17:29:02.725696 1070 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2e178ec907c4353af49d4f658f32c9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e178ec907c4353af49d4f658f32c9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33537 } }
I20260923 17:29:02.725911 1070 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2e178ec907c4353af49d4f658f32c9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:29:02.725993 1070 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2e178ec907c4353af49d4f658f32c9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2e178ec907c4353af49d4f658f32c9b, State: Initialized, Role: FOLLOWER
I20260923 17:29:02.726300 12180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:29:02.726496 1070 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2e178ec907c4353af49d4f658f32c9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e178ec907c4353af49d4f658f32c9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33537 } }
I20260923 17:29:02.726657 1070 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2e178ec907c4353af49d4f658f32c9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:29:02.726745 1070 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2e178ec907c4353af49d4f658f32c9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:29:02.726843 1070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2e178ec907c4353af49d4f658f32c9b [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:29:02.727838 1070 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2e178ec907c4353af49d4f658f32c9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e178ec907c4353af49d4f658f32c9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33537 } }
I20260923 17:29:02.728178 1070 leader_election.cc:302] T 00000000000000000000000000000000 P d2e178ec907c4353af49d4f658f32c9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2e178ec907c4353af49d4f658f32c9b; no voters:
I20260923 17:29:02.728534 1070 leader_election.cc:288] T 00000000000000000000000000000000 P d2e178ec907c4353af49d4f658f32c9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:29:02.728631 1079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2e178ec907c4353af49d4f658f32c9b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:29:02.728853 1079 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2e178ec907c4353af49d4f658f32c9b [term 1 LEADER]: Becoming Leader. State: Replica: d2e178ec907c4353af49d4f658f32c9b, State: Running, Role: LEADER
I20260923 17:29:02.729187 1079 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2e178ec907c4353af49d4f658f32c9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e178ec907c4353af49d4f658f32c9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33537 } }
I20260923 17:29:02.729809 1070 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2e178ec907c4353af49d4f658f32c9b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:29:02.730842 1081 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2e178ec907c4353af49d4f658f32c9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2e178ec907c4353af49d4f658f32c9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e178ec907c4353af49d4f658f32c9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33537 } } }
I20260923 17:29:02.731007 1081 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2e178ec907c4353af49d4f658f32c9b [sys.catalog]: This master's current role is: LEADER
I20260923 17:29:02.731422 1089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:29:02.732076 1082 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2e178ec907c4353af49d4f658f32c9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2e178ec907c4353af49d4f658f32c9b. Latest consensus state: current_term: 1 leader_uuid: "d2e178ec907c4353af49d4f658f32c9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e178ec907c4353af49d4f658f32c9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33537 } } }
I20260923 17:29:02.732299 1082 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2e178ec907c4353af49d4f658f32c9b [sys.catalog]: This master's current role is: LEADER
I20260923 17:29:02.734763 1089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:29:02.739468 1073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:02.728472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:44947 (local address 127.0.0.1:33537)
0923 17:29:02.728598 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:02.728604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:02.730502 (+ 1898us) server_negotiation.cc:407] Connection header received
0923 17:29:02.730752 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:02.730758 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:02.730824 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:02.730919 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:02.731922 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.732706 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.733706 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.733959 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.734068 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:02.735905 (+ 1837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:02.735909 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:02.735912 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:02.735939 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:02.737653 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:02.738169 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:02.738173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:02.738175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:02.738260 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:02.738594 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:02.738597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:02.738599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:02.738782 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:02.738947 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:02.739216 (+ 269us) server_negotiation.cc:299] Negotiation successful
0923 17:29:02.739312 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 17:29:02.741945 1089 catalog_manager.cc:1347] Generated new cluster ID: 5a264f78902b4bfb9c3849161496e4d8
I20260923 17:29:02.742050 1089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:29:02.748260 1089 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:29:02.749152 1089 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:29:02.758018 1089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2e178ec907c4353af49d4f658f32c9b: Generated new TSK 0
I20260923 17:29:02.758818 1089 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:29:02.766463 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 1002
W20260923 17:29:02.772136 987 connection.cc:535] client connection to 127.0.0.1:33537 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33537): Connection reset by peer (error 104)
2026-09-23T17:29:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1413 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:29:02 dist-test-slave-ptl9 krb5kdc[1101](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:29:02 dist-test-slave-ptl9 krb5kdc[1101](info): set up 2 sockets
Sep 23 17:29:02 dist-test-slave-ptl9 krb5kdc[1101](info): commencing operation
krb5kdc: starting...
W20260923 17:29:03.801661 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:29:03 dist-test-slave-ptl9 krb5kdc[1101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:29:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:29:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:29:03.924825 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36469
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:29:04.038738 1114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:04.039073 1114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:04.039162 1114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:29:04.043341 1114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:29:04.043440 1114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:04.043479 1114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:04.043536 1114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:29:04.043573 1114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:29:04.049038 1114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36469
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:29:04.050264 1114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:04.051337 1114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:04.059964 1122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:04.060055 1114 server_base.cc:1034] running on GCE node
W20260923 17:29:04.059978 1120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:04.059978 1119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:04.060818 1114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:04.062007 1114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:04.063228 1114 hybrid_clock.cc:648] HybridClock initialized: now 1790184544063205 us; error 35 us; skew 500 ppm
Sep 23 17:29:04 dist-test-slave-ptl9 krb5kdc[1101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:29:04.071254 1114 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:29:04.072254 1114 webserver.cc:460] Webserver started at http://127.0.0.1:41147/ using document root <none> and password file <none>
I20260923 17:29:04.072810 1114 fs_manager.cc:359] Metadata directory not provided
I20260923 17:29:04.072872 1114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:29:04.073040 1114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:29:04.075167 1114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "86d3c60a067e43aaa4827d291b466c5d"
format_stamp: "Formatted at 2026-09-23 17:29:04 on dist-test-slave-ptl9"
server_key: "14549ac87a582733c8707d88a11f1b41"
server_key_iv: "85c294db29d91717bb2e9ef93824a2a0"
server_key_version: "encryptionkey@0"
I20260923 17:29:04.075680 1114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "86d3c60a067e43aaa4827d291b466c5d"
format_stamp: "Formatted at 2026-09-23 17:29:04 on dist-test-slave-ptl9"
server_key: "14549ac87a582733c8707d88a11f1b41"
server_key_iv: "85c294db29d91717bb2e9ef93824a2a0"
server_key_version: "encryptionkey@0"
I20260923 17:29:04.079463 1114 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:29:04.081943 1129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:04.083228 1114 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:29:04.083338 1114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "86d3c60a067e43aaa4827d291b466c5d"
format_stamp: "Formatted at 2026-09-23 17:29:04 on dist-test-slave-ptl9"
server_key: "14549ac87a582733c8707d88a11f1b41"
server_key_iv: "85c294db29d91717bb2e9ef93824a2a0"
server_key_version: "encryptionkey@0"
I20260923 17:29:04.083429 1114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:29:04.111663 1114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:29:04.115057 1114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:29:04.115234 1114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:29:04.122757 1114 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36469
I20260923 17:29:04.122782 1181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36469 every 8 connection(s)
I20260923 17:29:04.123776 1114 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:29:04.126752 1182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:29:04.132251 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 1114
I20260923 17:29:04.132409 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:29:04.132396 1182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86d3c60a067e43aaa4827d291b466c5d: Bootstrap starting.
I20260923 17:29:04.132732 12180 external_mini_cluster.cc:1428] Setting key 3e7eb0e250720d19e25a57a28b35316b
I20260923 17:29:04.135130 1182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86d3c60a067e43aaa4827d291b466c5d: Neither blocks nor log segments found. Creating new log.
I20260923 17:29:04.135883 1182 log.cc:824] T 00000000000000000000000000000000 P 86d3c60a067e43aaa4827d291b466c5d: Log is configured to *not* fsync() on all Append() calls
I20260923 17:29:04.138137 1182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86d3c60a067e43aaa4827d291b466c5d: No bootstrap required, opened a new log
I20260923 17:29:04.142448 1185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:04.134221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:51961 (local address 127.0.0.1:36469)
0923 17:29:04.134700 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:04.134716 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:04.134752 (+ 36us) server_negotiation.cc:407] Connection header received
0923 17:29:04.135469 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:04.135488 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:04.135846 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:04.136170 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:04.136621 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:04.137749 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:04.138979 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:04.139486 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:04.139633 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:04.140555 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:04.140560 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:04.140573 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:29:04.141003 (+ 430us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:04.141058 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:29:04.141082 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:04.141191 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:04.141454 (+ 263us) server_negotiation.cc:299] Negotiation successful
0923 17:29:04.141633 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":138,"threads_started":1}
I20260923 17:29:04.142567 1182 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86d3c60a067e43aaa4827d291b466c5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86d3c60a067e43aaa4827d291b466c5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36469 } }
I20260923 17:29:04.142885 1182 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86d3c60a067e43aaa4827d291b466c5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:29:04.142968 1182 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86d3c60a067e43aaa4827d291b466c5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86d3c60a067e43aaa4827d291b466c5d, State: Initialized, Role: FOLLOWER
I20260923 17:29:04.143402 1182 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86d3c60a067e43aaa4827d291b466c5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86d3c60a067e43aaa4827d291b466c5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36469 } }
I20260923 17:29:04.143556 1182 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86d3c60a067e43aaa4827d291b466c5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:29:04.143627 1182 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86d3c60a067e43aaa4827d291b466c5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:29:04.143754 1182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86d3c60a067e43aaa4827d291b466c5d [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:29:04.144809 1182 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86d3c60a067e43aaa4827d291b466c5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86d3c60a067e43aaa4827d291b466c5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36469 } }
I20260923 17:29:04.145179 1182 leader_election.cc:302] T 00000000000000000000000000000000 P 86d3c60a067e43aaa4827d291b466c5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86d3c60a067e43aaa4827d291b466c5d; no voters:
I20260923 17:29:04.145576 1182 leader_election.cc:288] T 00000000000000000000000000000000 P 86d3c60a067e43aaa4827d291b466c5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:29:04.145638 1187 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86d3c60a067e43aaa4827d291b466c5d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:29:04.145699 12180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:29:04.145890 1187 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86d3c60a067e43aaa4827d291b466c5d [term 1 LEADER]: Becoming Leader. State: Replica: 86d3c60a067e43aaa4827d291b466c5d, State: Running, Role: LEADER
I20260923 17:29:04.146210 1187 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86d3c60a067e43aaa4827d291b466c5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86d3c60a067e43aaa4827d291b466c5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36469 } }
I20260923 17:29:04.146986 1182 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86d3c60a067e43aaa4827d291b466c5d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:29:04.147408 1189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86d3c60a067e43aaa4827d291b466c5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86d3c60a067e43aaa4827d291b466c5d. Latest consensus state: current_term: 1 leader_uuid: "86d3c60a067e43aaa4827d291b466c5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86d3c60a067e43aaa4827d291b466c5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36469 } } }
I20260923 17:29:04.147534 1189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86d3c60a067e43aaa4827d291b466c5d [sys.catalog]: This master's current role is: LEADER
I20260923 17:29:04.147872 1188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86d3c60a067e43aaa4827d291b466c5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86d3c60a067e43aaa4827d291b466c5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86d3c60a067e43aaa4827d291b466c5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36469 } } }
I20260923 17:29:04.147976 1188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86d3c60a067e43aaa4827d291b466c5d [sys.catalog]: This master's current role is: LEADER
I20260923 17:29:04.147998 1195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:29:04.151533 1195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:29:04.154292 1185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:04.148314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:48835 (local address 127.0.0.1:36469)
0923 17:29:04.148442 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:04.148447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:04.148824 (+ 377us) server_negotiation.cc:407] Connection header received
0923 17:29:04.149025 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:04.149029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:04.149079 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:04.149129 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:04.149631 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:04.150377 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:04.152436 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:04.152679 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:04.152795 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:04.153736 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:04.153740 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:04.153743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:29:04.153757 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:04.153786 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:29:04.153800 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:04.153908 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:04.154097 (+ 189us) server_negotiation.cc:299] Negotiation successful
0923 17:29:04.154135 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 17:29:04.157492 1195 catalog_manager.cc:1347] Generated new cluster ID: 0dd38140c87d4a1485c2f8f1423ae700
I20260923 17:29:04.157559 1195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:29:04.183110 1195 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:29:04.184134 1195 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:29:04.191138 1195 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86d3c60a067e43aaa4827d291b466c5d: Generated new TSK 0
I20260923 17:29:04.191912 1195 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:29:04.200960 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 1114
W20260923 17:29:04.206643 1099 connection.cc:535] client connection to 127.0.0.1:36469 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36469): Connection reset by peer (error 104)
2026-09-23T17:29:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1434 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:29:04 dist-test-slave-ptl9 krb5kdc[1213](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:29:04 dist-test-slave-ptl9 krb5kdc[1213](info): set up 2 sockets
Sep 23 17:29:04 dist-test-slave-ptl9 krb5kdc[1213](info): commencing operation
krb5kdc: starting...
W20260923 17:29:05.237643 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:29:05 dist-test-slave-ptl9 krb5kdc[1213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:29:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:29:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:29:05.360112 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46393
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:29:05.470377 1226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:05.470628 1226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:05.470669 1226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:29:05.474537 1226 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:29:05.474591 1226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:05.474609 1226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:05.474627 1226 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:29:05.474641 1226 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:29:05.479657 1226 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46393
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184271597484-12180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:29:05.480630 1226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:05.481599 1226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:05.490700 1232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:05.490698 1231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:05.490916 1226 server_base.cc:1034] running on GCE node
W20260923 17:29:05.490752 1234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:05.491567 1226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:05.492681 1226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:05.493856 1226 hybrid_clock.cc:648] HybridClock initialized: now 1790184545493831 us; error 32 us; skew 500 ppm
Sep 23 17:29:05 dist-test-slave-ptl9 krb5kdc[1213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184545, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:29:05.496794 1226 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:29:05.497787 1226 webserver.cc:460] Webserver started at http://127.0.0.1:38483/ using document root <none> and password file <none>
I20260923 17:29:05.498394 1226 fs_manager.cc:359] Metadata directory not provided
I20260923 17:29:05.498490 1226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:29:05.498728 1226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:29:05.500924 1226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "8aedd6f9cabd46da93aaeb0fbe2a8d9f"
format_stamp: "Formatted at 2026-09-23 17:29:05 on dist-test-slave-ptl9"
server_key: "ec679a9bb1d49325bc5b6bac1e67a45b"
server_key_iv: "54c2be8763bd82438cc541ed8e55d9a1"
server_key_version: "encryptionkey@0"
I20260923 17:29:05.501544 1226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "8aedd6f9cabd46da93aaeb0fbe2a8d9f"
format_stamp: "Formatted at 2026-09-23 17:29:05 on dist-test-slave-ptl9"
server_key: "ec679a9bb1d49325bc5b6bac1e67a45b"
server_key_iv: "54c2be8763bd82438cc541ed8e55d9a1"
server_key_version: "encryptionkey@0"
I20260923 17:29:05.505522 1226 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:29:05.508064 1241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:05.509537 1226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:29:05.509688 1226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "8aedd6f9cabd46da93aaeb0fbe2a8d9f"
format_stamp: "Formatted at 2026-09-23 17:29:05 on dist-test-slave-ptl9"
server_key: "ec679a9bb1d49325bc5b6bac1e67a45b"
server_key_iv: "54c2be8763bd82438cc541ed8e55d9a1"
server_key_version: "encryptionkey@0"
I20260923 17:29:05.509811 1226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:29:05.514513 1226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:29:05.517838 1226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:29:05.518033 1226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:29:05.525106 1226 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46393
I20260923 17:29:05.525133 1293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46393 every 8 connection(s)
I20260923 17:29:05.526147 1226 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:29:05.528820 1294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:29:05.534312 1294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8aedd6f9cabd46da93aaeb0fbe2a8d9f: Bootstrap starting.
I20260923 17:29:05.536000 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 1226
I20260923 17:29:05.536134 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:29:05.536451 12180 external_mini_cluster.cc:1428] Setting key c64db0b19bfeb90f96714186344d8e71
I20260923 17:29:05.536880 1294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8aedd6f9cabd46da93aaeb0fbe2a8d9f: Neither blocks nor log segments found. Creating new log.
I20260923 17:29:05.537750 1294 log.cc:824] T 00000000000000000000000000000000 P 8aedd6f9cabd46da93aaeb0fbe2a8d9f: Log is configured to *not* fsync() on all Append() calls
I20260923 17:29:05.539891 1294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8aedd6f9cabd46da93aaeb0fbe2a8d9f: No bootstrap required, opened a new log
I20260923 17:29:05.541787 1297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:05.537599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:46283 (local address 127.0.0.1:46393)
0923 17:29:05.538141 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:05.538161 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:05.538205 (+ 44us) server_negotiation.cc:407] Connection header received
0923 17:29:05.539026 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:05.539050 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:05.539381 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:05.539727 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:05.540416 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:05.540421 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:05.540435 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:29:05.540448 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:05.540506 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:29:05.540531 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:05.540638 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:05.540725 (+ 87us) server_negotiation.cc:299] Negotiation successful
0923 17:29:05.540872 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":182,"threads_started":1}
I20260923 17:29:05.543998 1294 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8aedd6f9cabd46da93aaeb0fbe2a8d9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aedd6f9cabd46da93aaeb0fbe2a8d9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46393 } }
I20260923 17:29:05.544303 1294 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8aedd6f9cabd46da93aaeb0fbe2a8d9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:29:05.544404 1294 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8aedd6f9cabd46da93aaeb0fbe2a8d9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8aedd6f9cabd46da93aaeb0fbe2a8d9f, State: Initialized, Role: FOLLOWER
I20260923 17:29:05.544546 12180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:29:05.544873 1294 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8aedd6f9cabd46da93aaeb0fbe2a8d9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aedd6f9cabd46da93aaeb0fbe2a8d9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46393 } }
I20260923 17:29:05.545028 1294 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8aedd6f9cabd46da93aaeb0fbe2a8d9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:29:05.545115 1294 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8aedd6f9cabd46da93aaeb0fbe2a8d9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:29:05.545238 1294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8aedd6f9cabd46da93aaeb0fbe2a8d9f [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:29:05.546237 1294 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8aedd6f9cabd46da93aaeb0fbe2a8d9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aedd6f9cabd46da93aaeb0fbe2a8d9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46393 } }
I20260923 17:29:05.546576 1294 leader_election.cc:302] T 00000000000000000000000000000000 P 8aedd6f9cabd46da93aaeb0fbe2a8d9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8aedd6f9cabd46da93aaeb0fbe2a8d9f; no voters:
I20260923 17:29:05.546919 1294 leader_election.cc:288] T 00000000000000000000000000000000 P 8aedd6f9cabd46da93aaeb0fbe2a8d9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:29:05.547010 1304 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8aedd6f9cabd46da93aaeb0fbe2a8d9f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:29:05.547238 1304 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8aedd6f9cabd46da93aaeb0fbe2a8d9f [term 1 LEADER]: Becoming Leader. State: Replica: 8aedd6f9cabd46da93aaeb0fbe2a8d9f, State: Running, Role: LEADER
I20260923 17:29:05.547567 1304 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8aedd6f9cabd46da93aaeb0fbe2a8d9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aedd6f9cabd46da93aaeb0fbe2a8d9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46393 } }
I20260923 17:29:05.548183 1294 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8aedd6f9cabd46da93aaeb0fbe2a8d9f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:29:05.548529 1297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:05.546755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:54305 (local address 127.0.0.1:46393)
0923 17:29:05.546891 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:05.546895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:05.547045 (+ 150us) server_negotiation.cc:407] Connection header received
0923 17:29:05.547178 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:05.547183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:05.547218 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:05.547286 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:05.548099 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:05.548103 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:05.548105 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:29:05.548117 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:05.548143 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:29:05.548157 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:05.548250 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:05.548350 (+ 100us) server_negotiation.cc:299] Negotiation successful
0923 17:29:05.548394 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 17:29:05.549417 1305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8aedd6f9cabd46da93aaeb0fbe2a8d9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8aedd6f9cabd46da93aaeb0fbe2a8d9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aedd6f9cabd46da93aaeb0fbe2a8d9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46393 } } }
I20260923 17:29:05.549559 1305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8aedd6f9cabd46da93aaeb0fbe2a8d9f [sys.catalog]: This master's current role is: LEADER
I20260923 17:29:05.549968 1313 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:29:05.550206 1306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8aedd6f9cabd46da93aaeb0fbe2a8d9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8aedd6f9cabd46da93aaeb0fbe2a8d9f. Latest consensus state: current_term: 1 leader_uuid: "8aedd6f9cabd46da93aaeb0fbe2a8d9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aedd6f9cabd46da93aaeb0fbe2a8d9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46393 } } }
I20260923 17:29:05.550343 1306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8aedd6f9cabd46da93aaeb0fbe2a8d9f [sys.catalog]: This master's current role is: LEADER
I20260923 17:29:05.553099 1313 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:29:05.558940 1313 catalog_manager.cc:1347] Generated new cluster ID: 1461efe349df41e68d7849fa4b6b4660
I20260923 17:29:05.559016 1313 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:29:05.566916 1313 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:29:05.567191 1313 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:29:05.578926 1313 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8aedd6f9cabd46da93aaeb0fbe2a8d9f: Generated new TSK 0
I20260923 17:29:05.579596 1313 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:29:05.584833 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 1226
2026-09-23T17:29:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1383 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:29:05 dist-test-slave-ptl9 krb5kdc[1325](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:29:05 dist-test-slave-ptl9 krb5kdc[1325](info): set up 2 sockets
Sep 23 17:29:05 dist-test-slave-ptl9 krb5kdc[1325](info): commencing operation
krb5kdc: starting...
W20260923 17:29:06.621737 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:29:06 dist-test-slave-ptl9 krb5kdc[1325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:29:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:29:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.229.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
I20260923 17:29:06.740736 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:34543
--webserver_interface=127.11.229.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:45409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:34543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:29:06.850579 1338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:06.850886 1338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:06.850963 1338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:29:06.854959 1338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:29:06.855058 1338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:06.855098 1338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:06.855134 1338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:29:06.855167 1338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:29:06.860321 1338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:45409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:34543
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:34543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:29:06.861407 1338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:06.862427 1338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:06.870975 1343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:06.871004 1346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:06.870965 1344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:06.871277 1338 server_base.cc:1034] running on GCE node
I20260923 17:29:06.871940 1338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:06.873034 1338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:06.874226 1338 hybrid_clock.cc:648] HybridClock initialized: now 1790184546874221 us; error 50 us; skew 500 ppm
Sep 23 17:29:06 dist-test-slave-ptl9 krb5kdc[1325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184546, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:29:06.876917 1338 init.cc:377] Logged in from keytab as kudu/127.11.229.62@KRBTEST.COM (short username kudu)
I20260923 17:29:06.877897 1338 webserver.cc:460] Webserver started at http://127.11.229.62:43607/ using document root <none> and password file <none>
I20260923 17:29:06.878497 1338 fs_manager.cc:359] Metadata directory not provided
I20260923 17:29:06.878592 1338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:29:06.878822 1338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:29:06.880852 1338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "b5426a02967f4ce9b6719674cf0d13d7"
format_stamp: "Formatted at 2026-09-23 17:29:06 on dist-test-slave-ptl9"
server_key: "5af70e805af1925af7266a31aae2d158"
server_key_iv: "4393a09d34351909f42e1fa805faf0e9"
server_key_version: "encryptionkey@0"
I20260923 17:29:06.881472 1338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "b5426a02967f4ce9b6719674cf0d13d7"
format_stamp: "Formatted at 2026-09-23 17:29:06 on dist-test-slave-ptl9"
server_key: "5af70e805af1925af7266a31aae2d158"
server_key_iv: "4393a09d34351909f42e1fa805faf0e9"
server_key_version: "encryptionkey@0"
I20260923 17:29:06.885313 1338 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:29:06.887864 1353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:06.889055 1338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:29:06.889201 1338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "b5426a02967f4ce9b6719674cf0d13d7"
format_stamp: "Formatted at 2026-09-23 17:29:06 on dist-test-slave-ptl9"
server_key: "5af70e805af1925af7266a31aae2d158"
server_key_iv: "4393a09d34351909f42e1fa805faf0e9"
server_key_version: "encryptionkey@0"
I20260923 17:29:06.889380 1338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:29:06.916386 1338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:29:06.919806 1338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:29:06.920025 1338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:29:06.927379 1338 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:34543
I20260923 17:29:06.927394 1405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:34543 every 8 connection(s)
I20260923 17:29:06.928309 1338 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:29:06.930968 1406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:29:06.936476 1406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5426a02967f4ce9b6719674cf0d13d7: Bootstrap starting.
I20260923 17:29:06.936676 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 1338
I20260923 17:29:06.936820 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:29:06.937114 12180 external_mini_cluster.cc:1428] Setting key 70dd24aa70dbb870dd0c401b80c8fb72
I20260923 17:29:06.939131 1406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5426a02967f4ce9b6719674cf0d13d7: Neither blocks nor log segments found. Creating new log.
I20260923 17:29:06.939839 1406 log.cc:824] T 00000000000000000000000000000000 P b5426a02967f4ce9b6719674cf0d13d7: Log is configured to *not* fsync() on all Append() calls
I20260923 17:29:06.942001 1406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5426a02967f4ce9b6719674cf0d13d7: No bootstrap required, opened a new log
Sep 23 17:29:06 dist-test-slave-ptl9 krb5kdc[1325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:29:06.946473 1406 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5426a02967f4ce9b6719674cf0d13d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5426a02967f4ce9b6719674cf0d13d7" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34543 } }
I20260923 17:29:06.946700 1406 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5426a02967f4ce9b6719674cf0d13d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:29:06.946767 1406 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5426a02967f4ce9b6719674cf0d13d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5426a02967f4ce9b6719674cf0d13d7, State: Initialized, Role: FOLLOWER
I20260923 17:29:06.947254 1406 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5426a02967f4ce9b6719674cf0d13d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5426a02967f4ce9b6719674cf0d13d7" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34543 } }
I20260923 17:29:06.947428 1406 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5426a02967f4ce9b6719674cf0d13d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:29:06.947499 1406 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5426a02967f4ce9b6719674cf0d13d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:29:06.947604 1406 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5426a02967f4ce9b6719674cf0d13d7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:29:06.948434 1406 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5426a02967f4ce9b6719674cf0d13d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5426a02967f4ce9b6719674cf0d13d7" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34543 } }
I20260923 17:29:06.948755 1406 leader_election.cc:302] T 00000000000000000000000000000000 P b5426a02967f4ce9b6719674cf0d13d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5426a02967f4ce9b6719674cf0d13d7; no voters:
I20260923 17:29:06.949085 1406 leader_election.cc:288] T 00000000000000000000000000000000 P b5426a02967f4ce9b6719674cf0d13d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:29:06.949150 1411 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5426a02967f4ce9b6719674cf0d13d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:29:06.949383 1411 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5426a02967f4ce9b6719674cf0d13d7 [term 1 LEADER]: Becoming Leader. State: Replica: b5426a02967f4ce9b6719674cf0d13d7, State: Running, Role: LEADER
I20260923 17:29:06.949703 1411 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5426a02967f4ce9b6719674cf0d13d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5426a02967f4ce9b6719674cf0d13d7" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34543 } }
I20260923 17:29:06.950390 1406 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5426a02967f4ce9b6719674cf0d13d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:29:06.951521 1412 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5426a02967f4ce9b6719674cf0d13d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5426a02967f4ce9b6719674cf0d13d7. Latest consensus state: current_term: 1 leader_uuid: "b5426a02967f4ce9b6719674cf0d13d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5426a02967f4ce9b6719674cf0d13d7" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34543 } } }
I20260923 17:29:06.951642 1412 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5426a02967f4ce9b6719674cf0d13d7 [sys.catalog]: This master's current role is: LEADER
I20260923 17:29:06.951793 1413 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5426a02967f4ce9b6719674cf0d13d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5426a02967f4ce9b6719674cf0d13d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5426a02967f4ce9b6719674cf0d13d7" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34543 } } }
I20260923 17:29:06.951964 1413 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5426a02967f4ce9b6719674cf0d13d7 [sys.catalog]: This master's current role is: LEADER
I20260923 17:29:06.952512 1409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:06.938475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56937 (local address 127.11.229.62:34543)
0923 17:29:06.938910 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:06.938925 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:06.938959 (+ 34us) server_negotiation.cc:407] Connection header received
0923 17:29:06.939619 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:06.939636 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:06.940033 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:06.940408 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:06.941189 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:06.942263 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:06.943409 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:06.943848 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:06.943963 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:06.946783 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:06.946790 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:06.946807 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:06.946841 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:06.948775 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:06.949340 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:06.949346 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:06.949388 (+ 42us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:06.949568 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:06.949970 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:06.949972 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:06.949974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:06.950383 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:06.950616 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:06.951251 (+ 635us) server_negotiation.cc:299] Negotiation successful
0923 17:29:06.951539 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":130,"threads_started":1}
I20260923 17:29:06.955883 12180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260923 17:29:06.956122 1426 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b5426a02967f4ce9b6719674cf0d13d7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:29:06.956202 1426 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:29:06.956266 1420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:29:06.957139 1420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:29:06.963382 1420 catalog_manager.cc:1347] Generated new cluster ID: b4ec2405ef984fa099a4fe70d1aa1985
I20260923 17:29:06.963526 1420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:29:06.966482 1409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:06.957955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48129 (local address 127.11.229.62:34543)
0923 17:29:06.958064 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:06.958069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:06.958261 (+ 192us) server_negotiation.cc:407] Connection header received
0923 17:29:06.958389 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:06.958392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:06.958440 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:06.958523 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:06.959209 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:06.959961 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:06.961075 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:06.961400 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:06.961466 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:06.963328 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:06.963332 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:06.963334 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:06.963358 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:06.964702 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:06.965176 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:06.965180 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:06.965181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:06.965310 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:06.965622 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:06.965625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:06.965627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:06.965797 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:06.965948 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:06.966223 (+ 275us) server_negotiation.cc:299] Negotiation successful
0923 17:29:06.966314 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 17:29:06.983198 1420 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:29:06.984513 1420 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:29:06.993218 1420 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5426a02967f4ce9b6719674cf0d13d7: Generated new TSK 0
I20260923 17:29:06.994045 1420 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:29:07.003464 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 1338
W20260923 17:29:07.010527 1323 connection.cc:535] client connection to 127.11.229.62:34543 recv error: Network error: failed to read from TLS socket (remote: 127.11.229.62:34543): Connection reset by peer (error 104)
2026-09-23T17:29:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1420 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:29:07 dist-test-slave-ptl9 krb5kdc[1437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:29:07 dist-test-slave-ptl9 krb5kdc[1437](info): set up 2 sockets
Sep 23 17:29:07 dist-test-slave-ptl9 krb5kdc[1437](info): commencing operation
krb5kdc: starting...
W20260923 17:29:08.037751 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:29:08 dist-test-slave-ptl9 krb5kdc[1437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:29:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:29:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.229.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
I20260923 17:29:08.160075 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:43109
--webserver_interface=127.11.229.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:39593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:43109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:29:08.269866 1450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:08.270134 1450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:08.270176 1450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:29:08.274099 1450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:29:08.274153 1450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:08.274173 1450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:08.274189 1450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:29:08.274204 1450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:29:08.279512 1450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:39593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:43109
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:43109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:29:08.280591 1450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:08.281617 1450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:08.290282 1456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:08.290298 1458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:08.290297 1455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:08.290566 1450 server_base.cc:1034] running on GCE node
I20260923 17:29:08.291172 1450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:08.292284 1450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:08.293431 1450 hybrid_clock.cc:648] HybridClock initialized: now 1790184548293404 us; error 38 us; skew 500 ppm
Sep 23 17:29:08 dist-test-slave-ptl9 krb5kdc[1437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184548, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:29:08.296305 1450 init.cc:377] Logged in from keytab as kudu/127.11.229.62@KRBTEST.COM (short username kudu)
I20260923 17:29:08.297312 1450 webserver.cc:460] Webserver started at http://127.11.229.62:44411/ using document root <none> and password file <none>
I20260923 17:29:08.297888 1450 fs_manager.cc:359] Metadata directory not provided
I20260923 17:29:08.297991 1450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:29:08.298223 1450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:29:08.300239 1450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "7a5e2d71ad48437fa0bd04bbf4199620"
format_stamp: "Formatted at 2026-09-23 17:29:08 on dist-test-slave-ptl9"
server_key: "2df93fe0bc50a40e40421a6dae03c9c0"
server_key_iv: "5e2395b51a43053f54e040717d4d87a2"
server_key_version: "encryptionkey@0"
I20260923 17:29:08.300796 1450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "7a5e2d71ad48437fa0bd04bbf4199620"
format_stamp: "Formatted at 2026-09-23 17:29:08 on dist-test-slave-ptl9"
server_key: "2df93fe0bc50a40e40421a6dae03c9c0"
server_key_iv: "5e2395b51a43053f54e040717d4d87a2"
server_key_version: "encryptionkey@0"
I20260923 17:29:08.304456 1450 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260923 17:29:08.306914 1465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:08.308140 1450 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:29:08.308282 1450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "7a5e2d71ad48437fa0bd04bbf4199620"
format_stamp: "Formatted at 2026-09-23 17:29:08 on dist-test-slave-ptl9"
server_key: "2df93fe0bc50a40e40421a6dae03c9c0"
server_key_iv: "5e2395b51a43053f54e040717d4d87a2"
server_key_version: "encryptionkey@0"
I20260923 17:29:08.308398 1450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:29:08.329042 1450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:29:08.332433 1450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:29:08.332654 1450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:29:08.340271 1450 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:43109
I20260923 17:29:08.340330 1517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:43109 every 8 connection(s)
I20260923 17:29:08.341477 1450 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:29:08.344369 1518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:29:08.346705 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 1450
I20260923 17:29:08.346859 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:29:08.347141 12180 external_mini_cluster.cc:1428] Setting key 07d315ca967a8e246a6830478429e3ea
I20260923 17:29:08.350517 1518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a5e2d71ad48437fa0bd04bbf4199620: Bootstrap starting.
I20260923 17:29:08.353045 1518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a5e2d71ad48437fa0bd04bbf4199620: Neither blocks nor log segments found. Creating new log.
I20260923 17:29:08.354004 1518 log.cc:824] T 00000000000000000000000000000000 P 7a5e2d71ad48437fa0bd04bbf4199620: Log is configured to *not* fsync() on all Append() calls
I20260923 17:29:08.356595 1518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a5e2d71ad48437fa0bd04bbf4199620: No bootstrap required, opened a new log
I20260923 17:29:08.360272 1521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:08.348470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47999 (local address 127.11.229.62:43109)
0923 17:29:08.348890 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:08.348905 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:08.348941 (+ 36us) server_negotiation.cc:407] Connection header received
0923 17:29:08.349639 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:08.349657 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:08.349999 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:08.350268 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:08.350685 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:08.351820 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:08.353082 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:08.353821 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:08.353981 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:08.357767 (+ 3786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:08.357775 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:08.357842 (+ 67us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:29:08.358498 (+ 656us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:08.358594 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:29:08.358626 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:08.358788 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:08.358984 (+ 196us) server_negotiation.cc:299] Negotiation successful
0923 17:29:08.359134 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":140,"threads_started":1}
I20260923 17:29:08.361685 1518 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a5e2d71ad48437fa0bd04bbf4199620 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5e2d71ad48437fa0bd04bbf4199620" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 43109 } }
I20260923 17:29:08.361910 1518 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a5e2d71ad48437fa0bd04bbf4199620 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:29:08.361984 1518 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a5e2d71ad48437fa0bd04bbf4199620 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a5e2d71ad48437fa0bd04bbf4199620, State: Initialized, Role: FOLLOWER
I20260923 17:29:08.362412 1518 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a5e2d71ad48437fa0bd04bbf4199620 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5e2d71ad48437fa0bd04bbf4199620" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 43109 } }
I20260923 17:29:08.362568 1518 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a5e2d71ad48437fa0bd04bbf4199620 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:29:08.362637 1518 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a5e2d71ad48437fa0bd04bbf4199620 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:29:08.362732 1518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a5e2d71ad48437fa0bd04bbf4199620 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:29:08.363651 1518 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a5e2d71ad48437fa0bd04bbf4199620 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5e2d71ad48437fa0bd04bbf4199620" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 43109 } }
I20260923 17:29:08.363673 12180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:29:08.363996 1518 leader_election.cc:302] T 00000000000000000000000000000000 P 7a5e2d71ad48437fa0bd04bbf4199620 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a5e2d71ad48437fa0bd04bbf4199620; no voters:
I20260923 17:29:08.364301 1518 leader_election.cc:288] T 00000000000000000000000000000000 P 7a5e2d71ad48437fa0bd04bbf4199620 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:29:08.364424 1523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a5e2d71ad48437fa0bd04bbf4199620 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:29:08.364656 1523 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a5e2d71ad48437fa0bd04bbf4199620 [term 1 LEADER]: Becoming Leader. State: Replica: 7a5e2d71ad48437fa0bd04bbf4199620, State: Running, Role: LEADER
I20260923 17:29:08.365011 1523 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a5e2d71ad48437fa0bd04bbf4199620 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5e2d71ad48437fa0bd04bbf4199620" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 43109 } }
I20260923 17:29:08.365658 1518 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a5e2d71ad48437fa0bd04bbf4199620 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:29:08.366290 1525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a5e2d71ad48437fa0bd04bbf4199620 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a5e2d71ad48437fa0bd04bbf4199620. Latest consensus state: current_term: 1 leader_uuid: "7a5e2d71ad48437fa0bd04bbf4199620" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5e2d71ad48437fa0bd04bbf4199620" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 43109 } } }
I20260923 17:29:08.366416 1525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a5e2d71ad48437fa0bd04bbf4199620 [sys.catalog]: This master's current role is: LEADER
I20260923 17:29:08.367080 1524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a5e2d71ad48437fa0bd04bbf4199620 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a5e2d71ad48437fa0bd04bbf4199620" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5e2d71ad48437fa0bd04bbf4199620" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 43109 } } }
I20260923 17:29:08.367182 1524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a5e2d71ad48437fa0bd04bbf4199620 [sys.catalog]: This master's current role is: LEADER
I20260923 17:29:08.367475 1535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:29:08.371270 1535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:29:08.374483 1521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:08.365940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52661 (local address 127.11.229.62:43109)
0923 17:29:08.366079 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:08.366086 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:08.366718 (+ 632us) server_negotiation.cc:407] Connection header received
0923 17:29:08.366946 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:08.366950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:08.366995 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:08.367044 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:08.367625 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:08.368420 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:08.369333 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:08.369581 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:08.369698 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:08.373922 (+ 4224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:08.373927 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:08.373930 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:29:08.373946 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:08.373979 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:29:08.373994 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:08.374090 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:08.374295 (+ 205us) server_negotiation.cc:299] Negotiation successful
0923 17:29:08.374336 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 17:29:08.377625 1535 catalog_manager.cc:1347] Generated new cluster ID: 57c8dedc9c894cd894e04cf6341011c2
I20260923 17:29:08.377704 1535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:29:08.386325 1535 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:29:08.387498 1535 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:29:08.392961 1535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a5e2d71ad48437fa0bd04bbf4199620: Generated new TSK 0
I20260923 17:29:08.393832 1535 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:29:08.405499 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 1450
W20260923 17:29:08.411484 1435 connection.cc:535] client connection to 127.11.229.62:43109 recv error: Network error: failed to read from TLS socket (remote: 127.11.229.62:43109): Connection reset by peer (error 104)
2026-09-23T17:29:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1400 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:29:08 dist-test-slave-ptl9 krb5kdc[1549](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:29:08 dist-test-slave-ptl9 krb5kdc[1549](info): set up 2 sockets
Sep 23 17:29:08 dist-test-slave-ptl9 krb5kdc[1549](info): commencing operation
krb5kdc: starting...
W20260923 17:29:09.449756 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:29:09 dist-test-slave-ptl9 krb5kdc[1549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:29:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:29:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.229.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
I20260923 17:29:09.572526 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:42435
--webserver_interface=127.11.229.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:40525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:42435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:29:09.711372 1562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:09.711663 1562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:09.711709 1562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:29:09.716032 1562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:29:09.716101 1562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:09.716122 1562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:09.716140 1562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:29:09.716187 1562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:29:09.721786 1562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:40525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:42435
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:42435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:29:09.722941 1562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:09.724010 1562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:09.733176 1568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:09.733326 1567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:09.733374 1562 server_base.cc:1034] running on GCE node
W20260923 17:29:09.733176 1570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:09.734061 1562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:09.735165 1562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:09.736325 1562 hybrid_clock.cc:648] HybridClock initialized: now 1790184549736307 us; error 31 us; skew 500 ppm
Sep 23 17:29:09 dist-test-slave-ptl9 krb5kdc[1549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184549, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:29:09.744213 1562 init.cc:377] Logged in from keytab as kudu/127.11.229.62@KRBTEST.COM (short username kudu)
I20260923 17:29:09.745350 1562 webserver.cc:460] Webserver started at http://127.11.229.62:35019/ using document root <none> and password file <none>
I20260923 17:29:09.745957 1562 fs_manager.cc:359] Metadata directory not provided
I20260923 17:29:09.746053 1562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:29:09.746302 1562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:29:09.748440 1562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "d5c37e7723e4403cacb6d7e1a6e71cb1"
format_stamp: "Formatted at 2026-09-23 17:29:09 on dist-test-slave-ptl9"
server_key: "66ceb3d88e7e5862167ac0d06b1a5f71"
server_key_iv: "6560fc2102acad4b7d424702b1816a8c"
server_key_version: "encryptionkey@0"
I20260923 17:29:09.749042 1562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "d5c37e7723e4403cacb6d7e1a6e71cb1"
format_stamp: "Formatted at 2026-09-23 17:29:09 on dist-test-slave-ptl9"
server_key: "66ceb3d88e7e5862167ac0d06b1a5f71"
server_key_iv: "6560fc2102acad4b7d424702b1816a8c"
server_key_version: "encryptionkey@0"
I20260923 17:29:09.752900 1562 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260923 17:29:09.755617 1577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:09.756897 1562 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:29:09.757045 1562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "d5c37e7723e4403cacb6d7e1a6e71cb1"
format_stamp: "Formatted at 2026-09-23 17:29:09 on dist-test-slave-ptl9"
server_key: "66ceb3d88e7e5862167ac0d06b1a5f71"
server_key_iv: "6560fc2102acad4b7d424702b1816a8c"
server_key_version: "encryptionkey@0"
I20260923 17:29:09.757164 1562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:29:09.761885 1562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:29:09.765177 1562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:29:09.765409 1562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:29:09.772559 1562 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:42435
I20260923 17:29:09.772570 1629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:42435 every 8 connection(s)
I20260923 17:29:09.773640 1562 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:29:09.776430 1630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:29:09.782029 1630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5c37e7723e4403cacb6d7e1a6e71cb1: Bootstrap starting.
I20260923 17:29:09.783078 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 1562
I20260923 17:29:09.783245 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:29:09.783540 12180 external_mini_cluster.cc:1428] Setting key 4ce499f2a45472483c50eafa4130755b
I20260923 17:29:09.784945 1630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5c37e7723e4403cacb6d7e1a6e71cb1: Neither blocks nor log segments found. Creating new log.
I20260923 17:29:09.785724 1630 log.cc:824] T 00000000000000000000000000000000 P d5c37e7723e4403cacb6d7e1a6e71cb1: Log is configured to *not* fsync() on all Append() calls
I20260923 17:29:09.787760 1630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5c37e7723e4403cacb6d7e1a6e71cb1: No bootstrap required, opened a new log
I20260923 17:29:09.789016 1633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:09.784691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33799 (local address 127.11.229.62:42435)
0923 17:29:09.785198 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:09.785213 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:09.785250 (+ 37us) server_negotiation.cc:407] Connection header received
0923 17:29:09.786032 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:09.786076 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:09.786448 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:09.786811 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:09.787606 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:09.787611 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:09.787623 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:29:09.787633 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:09.787684 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:29:09.787703 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:09.787824 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:09.787962 (+ 138us) server_negotiation.cc:299] Negotiation successful
0923 17:29:09.788118 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":149,"threads_started":1}
I20260923 17:29:09.791978 12180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:29:09.792166 1630 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5c37e7723e4403cacb6d7e1a6e71cb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c37e7723e4403cacb6d7e1a6e71cb1" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 42435 } }
I20260923 17:29:09.792384 1630 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5c37e7723e4403cacb6d7e1a6e71cb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:29:09.792451 1630 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5c37e7723e4403cacb6d7e1a6e71cb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5c37e7723e4403cacb6d7e1a6e71cb1, State: Initialized, Role: FOLLOWER
I20260923 17:29:09.792863 1630 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5c37e7723e4403cacb6d7e1a6e71cb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c37e7723e4403cacb6d7e1a6e71cb1" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 42435 } }
I20260923 17:29:09.792991 1630 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5c37e7723e4403cacb6d7e1a6e71cb1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:29:09.793071 1630 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5c37e7723e4403cacb6d7e1a6e71cb1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:29:09.793183 1630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5c37e7723e4403cacb6d7e1a6e71cb1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:29:09.794205 1630 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5c37e7723e4403cacb6d7e1a6e71cb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c37e7723e4403cacb6d7e1a6e71cb1" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 42435 } }
I20260923 17:29:09.794601 1630 leader_election.cc:302] T 00000000000000000000000000000000 P d5c37e7723e4403cacb6d7e1a6e71cb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5c37e7723e4403cacb6d7e1a6e71cb1; no voters:
I20260923 17:29:09.794968 1630 leader_election.cc:288] T 00000000000000000000000000000000 P d5c37e7723e4403cacb6d7e1a6e71cb1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:29:09.795011 1640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5c37e7723e4403cacb6d7e1a6e71cb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:29:09.795188 1640 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5c37e7723e4403cacb6d7e1a6e71cb1 [term 1 LEADER]: Becoming Leader. State: Replica: d5c37e7723e4403cacb6d7e1a6e71cb1, State: Running, Role: LEADER
I20260923 17:29:09.795495 1640 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5c37e7723e4403cacb6d7e1a6e71cb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c37e7723e4403cacb6d7e1a6e71cb1" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 42435 } }
I20260923 17:29:09.796278 1630 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5c37e7723e4403cacb6d7e1a6e71cb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:29:09.796413 1633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:09.794330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60693 (local address 127.11.229.62:42435)
0923 17:29:09.794454 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:09.794458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:09.794620 (+ 162us) server_negotiation.cc:407] Connection header received
0923 17:29:09.794707 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:09.794710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:09.794746 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:09.794832 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:09.795879 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:09.795883 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:09.795885 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:29:09.795897 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:09.795924 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:29:09.795935 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:09.796008 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:09.796167 (+ 159us) server_negotiation.cc:299] Negotiation successful
0923 17:29:09.796206 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 17:29:09.797186 1642 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5c37e7723e4403cacb6d7e1a6e71cb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5c37e7723e4403cacb6d7e1a6e71cb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c37e7723e4403cacb6d7e1a6e71cb1" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 42435 } } }
I20260923 17:29:09.797246 1641 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5c37e7723e4403cacb6d7e1a6e71cb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5c37e7723e4403cacb6d7e1a6e71cb1. Latest consensus state: current_term: 1 leader_uuid: "d5c37e7723e4403cacb6d7e1a6e71cb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c37e7723e4403cacb6d7e1a6e71cb1" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 42435 } } }
I20260923 17:29:09.797422 1642 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5c37e7723e4403cacb6d7e1a6e71cb1 [sys.catalog]: This master's current role is: LEADER
I20260923 17:29:09.797431 1641 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5c37e7723e4403cacb6d7e1a6e71cb1 [sys.catalog]: This master's current role is: LEADER
I20260923 17:29:09.798774 1649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:29:09.801927 1649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:29:09.807670 1649 catalog_manager.cc:1347] Generated new cluster ID: 901ebfa31a1d4ba190016099cbb66fc4
I20260923 17:29:09.807746 1649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:29:09.815374 1649 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:29:09.815694 1649 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:29:09.823474 1649 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5c37e7723e4403cacb6d7e1a6e71cb1: Generated new TSK 0
I20260923 17:29:09.824172 1649 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:29:09.831746 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 1562
2026-09-23T17:29:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1426 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:29:09 dist-test-slave-ptl9 krb5kdc[1661](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:29:09 dist-test-slave-ptl9 krb5kdc[1661](info): set up 2 sockets
Sep 23 17:29:09 dist-test-slave-ptl9 krb5kdc[1661](info): commencing operation
krb5kdc: starting...
W20260923 17:29:10.865680 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:29:10 dist-test-slave-ptl9 krb5kdc[1661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:29:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:29:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.229.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
I20260923 17:29:10.984443 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:39277
--webserver_interface=127.11.229.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:34475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:39277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:29:11.095329 1674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:11.095650 1674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:11.095733 1674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:29:11.099819 1674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:29:11.099916 1674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:11.099958 1674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:11.100014 1674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:29:11.100052 1674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:29:11.105597 1674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:34475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:39277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:39277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:29:11.106700 1674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:11.107755 1674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:11.116405 1679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:11.116459 1674 server_base.cc:1034] running on GCE node
W20260923 17:29:11.116411 1682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:11.116400 1680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:11.117169 1674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:11.118350 1674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:11.119536 1674 hybrid_clock.cc:648] HybridClock initialized: now 1790184551119509 us; error 40 us; skew 500 ppm
Sep 23 17:29:11 dist-test-slave-ptl9 krb5kdc[1661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184551, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:29:11.122416 1674 init.cc:377] Logged in from keytab as kudu/127.11.229.62@KRBTEST.COM (short username kudu)
I20260923 17:29:11.123377 1674 webserver.cc:460] Webserver started at http://127.11.229.62:38331/ using document root <none> and password file <none>
I20260923 17:29:11.124002 1674 fs_manager.cc:359] Metadata directory not provided
I20260923 17:29:11.124104 1674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:29:11.124372 1674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:29:11.126622 1674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "6c0f617c52a74a37a6dcd36e9da771ae"
format_stamp: "Formatted at 2026-09-23 17:29:11 on dist-test-slave-ptl9"
server_key: "97074142cd639d3f3e633493234c6a92"
server_key_iv: "3d14a64af145ac31c41fea73da3061a9"
server_key_version: "encryptionkey@0"
I20260923 17:29:11.127262 1674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "6c0f617c52a74a37a6dcd36e9da771ae"
format_stamp: "Formatted at 2026-09-23 17:29:11 on dist-test-slave-ptl9"
server_key: "97074142cd639d3f3e633493234c6a92"
server_key_iv: "3d14a64af145ac31c41fea73da3061a9"
server_key_version: "encryptionkey@0"
I20260923 17:29:11.131263 1674 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:29:11.133872 1689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:11.135143 1674 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:29:11.135306 1674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "6c0f617c52a74a37a6dcd36e9da771ae"
format_stamp: "Formatted at 2026-09-23 17:29:11 on dist-test-slave-ptl9"
server_key: "97074142cd639d3f3e633493234c6a92"
server_key_iv: "3d14a64af145ac31c41fea73da3061a9"
server_key_version: "encryptionkey@0"
I20260923 17:29:11.135432 1674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:29:11.146147 1674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:29:11.149668 1674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:29:11.149919 1674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:29:11.157714 1674 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:39277
I20260923 17:29:11.157729 1741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:39277 every 8 connection(s)
I20260923 17:29:11.158746 1674 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:29:11.160166 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 1674
I20260923 17:29:11.160339 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:29:11.160641 12180 external_mini_cluster.cc:1428] Setting key bd2d6b68e749b71514491eb9096640b8
I20260923 17:29:11.162065 1742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:29:11.168879 1742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c0f617c52a74a37a6dcd36e9da771ae: Bootstrap starting.
Sep 23 17:29:11 dist-test-slave-ptl9 krb5kdc[1661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:29:11.171525 1742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c0f617c52a74a37a6dcd36e9da771ae: Neither blocks nor log segments found. Creating new log.
I20260923 17:29:11.172238 1742 log.cc:824] T 00000000000000000000000000000000 P 6c0f617c52a74a37a6dcd36e9da771ae: Log is configured to *not* fsync() on all Append() calls
I20260923 17:29:11.174475 1742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c0f617c52a74a37a6dcd36e9da771ae: No bootstrap required, opened a new log
I20260923 17:29:11.176265 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:11.162078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58051 (local address 127.11.229.62:39277)
0923 17:29:11.162570 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:11.162589 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:11.162631 (+ 42us) server_negotiation.cc:407] Connection header received
0923 17:29:11.163505 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:11.163528 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:11.163957 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:11.164308 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:11.165077 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:11.166599 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:11.167518 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:11.168223 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:11.168378 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:11.171284 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:11.171292 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:11.171306 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:11.171336 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:11.173229 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:11.173807 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:11.173812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:11.173817 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:11.173932 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:11.174222 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:11.174224 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:11.174225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:11.174684 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:11.174858 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:11.175151 (+ 293us) server_negotiation.cc:299] Negotiation successful
0923 17:29:11.175490 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":151,"threads_started":1}
I20260923 17:29:11.178545 1742 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c0f617c52a74a37a6dcd36e9da771ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0f617c52a74a37a6dcd36e9da771ae" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 39277 } }
I20260923 17:29:11.178815 1742 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c0f617c52a74a37a6dcd36e9da771ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:29:11.178880 1742 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c0f617c52a74a37a6dcd36e9da771ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c0f617c52a74a37a6dcd36e9da771ae, State: Initialized, Role: FOLLOWER
I20260923 17:29:11.179353 1742 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c0f617c52a74a37a6dcd36e9da771ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0f617c52a74a37a6dcd36e9da771ae" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 39277 } }
I20260923 17:29:11.179443 12180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:29:11.179490 1742 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c0f617c52a74a37a6dcd36e9da771ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:29:11.179569 1742 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c0f617c52a74a37a6dcd36e9da771ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:29:11.179665 1742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c0f617c52a74a37a6dcd36e9da771ae [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:29:11.180595 1742 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c0f617c52a74a37a6dcd36e9da771ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0f617c52a74a37a6dcd36e9da771ae" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 39277 } }
I20260923 17:29:11.181006 1742 leader_election.cc:302] T 00000000000000000000000000000000 P 6c0f617c52a74a37a6dcd36e9da771ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c0f617c52a74a37a6dcd36e9da771ae; no voters:
I20260923 17:29:11.181368 1742 leader_election.cc:288] T 00000000000000000000000000000000 P 6c0f617c52a74a37a6dcd36e9da771ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:29:11.181483 1751 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c0f617c52a74a37a6dcd36e9da771ae [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:29:11.181748 1751 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c0f617c52a74a37a6dcd36e9da771ae [term 1 LEADER]: Becoming Leader. State: Replica: 6c0f617c52a74a37a6dcd36e9da771ae, State: Running, Role: LEADER
I20260923 17:29:11.182082 1751 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c0f617c52a74a37a6dcd36e9da771ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0f617c52a74a37a6dcd36e9da771ae" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 39277 } }
I20260923 17:29:11.182777 1742 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c0f617c52a74a37a6dcd36e9da771ae [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:29:11.183322 1751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c0f617c52a74a37a6dcd36e9da771ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c0f617c52a74a37a6dcd36e9da771ae. Latest consensus state: current_term: 1 leader_uuid: "6c0f617c52a74a37a6dcd36e9da771ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0f617c52a74a37a6dcd36e9da771ae" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 39277 } } }
I20260923 17:29:11.183455 1751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c0f617c52a74a37a6dcd36e9da771ae [sys.catalog]: This master's current role is: LEADER
I20260923 17:29:11.183905 1757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:29:11.183928 1752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c0f617c52a74a37a6dcd36e9da771ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c0f617c52a74a37a6dcd36e9da771ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0f617c52a74a37a6dcd36e9da771ae" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 39277 } } }
I20260923 17:29:11.184144 1752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c0f617c52a74a37a6dcd36e9da771ae [sys.catalog]: This master's current role is: LEADER
I20260923 17:29:11.187871 1757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:29:11.192378 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:11.181919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37201 (local address 127.11.229.62:39277)
0923 17:29:11.182043 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:11.182047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:11.182156 (+ 109us) server_negotiation.cc:407] Connection header received
0923 17:29:11.182336 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:11.182339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:11.182386 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:11.182436 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:11.183363 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:11.184228 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:11.185563 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:11.185920 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:11.186000 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:11.187807 (+ 1807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:11.187811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:11.187815 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:11.187844 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:11.189994 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:11.190514 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:11.190517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:11.190520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:11.190611 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:11.190939 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:11.190942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:11.190944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:11.191169 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:11.191308 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:11.192093 (+ 785us) server_negotiation.cc:299] Negotiation successful
0923 17:29:11.192216 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 17:29:11.194228 1757 catalog_manager.cc:1347] Generated new cluster ID: d3fec56030174c66bb21f792c42ab41a
I20260923 17:29:11.194352 1757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:29:11.214169 1757 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:29:11.215266 1757 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:29:11.221923 1757 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c0f617c52a74a37a6dcd36e9da771ae: Generated new TSK 0
I20260923 17:29:11.222709 1757 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:29:11.232450 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 1674
2026-09-23T17:29:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1399 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:29:11 dist-test-slave-ptl9 krb5kdc[1773](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:29:11 dist-test-slave-ptl9 krb5kdc[1773](info): set up 2 sockets
Sep 23 17:29:11 dist-test-slave-ptl9 krb5kdc[1773](info): commencing operation
krb5kdc: starting...
W20260923 17:29:12.313907 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:29:12 dist-test-slave-ptl9 krb5kdc[1773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:29:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:29:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.229.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
I20260923 17:29:12.440110 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:34749
--webserver_interface=127.11.229.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:40111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:34749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:29:12.552920 1786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:12.553220 1786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:12.553333 1786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:29:12.557286 1786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:29:12.557380 1786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:12.557418 1786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:12.557477 1786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:29:12.557515 1786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:29:12.562690 1786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:40111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:34749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:34749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:29:12.563784 1786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:12.564807 1786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:12.573218 1794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:12.573354 1792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:12.573298 1791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:12.573450 1786 server_base.cc:1034] running on GCE node
I20260923 17:29:12.574163 1786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:12.575230 1786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:12.576421 1786 hybrid_clock.cc:648] HybridClock initialized: now 1790184552576410 us; error 30 us; skew 500 ppm
Sep 23 17:29:12 dist-test-slave-ptl9 krb5kdc[1773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184552, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:29:12.579214 1786 init.cc:377] Logged in from keytab as kudu/127.11.229.62@KRBTEST.COM (short username kudu)
I20260923 17:29:12.580140 1786 webserver.cc:460] Webserver started at http://127.11.229.62:46267/ using document root <none> and password file <none>
I20260923 17:29:12.580705 1786 fs_manager.cc:359] Metadata directory not provided
I20260923 17:29:12.580797 1786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:29:12.581039 1786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:29:12.583138 1786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "b574b94cfef04764ab1c9ba1768dbcb9"
format_stamp: "Formatted at 2026-09-23 17:29:12 on dist-test-slave-ptl9"
server_key: "84fc4e7f72e710bebed4d045a4559e50"
server_key_iv: "e22c3f9a61310a94c1124efc178176ba"
server_key_version: "encryptionkey@0"
I20260923 17:29:12.583706 1786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "b574b94cfef04764ab1c9ba1768dbcb9"
format_stamp: "Formatted at 2026-09-23 17:29:12 on dist-test-slave-ptl9"
server_key: "84fc4e7f72e710bebed4d045a4559e50"
server_key_iv: "e22c3f9a61310a94c1124efc178176ba"
server_key_version: "encryptionkey@0"
I20260923 17:29:12.587483 1786 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 17:29:12.589917 1801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:12.591178 1786 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:29:12.591358 1786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "b574b94cfef04764ab1c9ba1768dbcb9"
format_stamp: "Formatted at 2026-09-23 17:29:12 on dist-test-slave-ptl9"
server_key: "84fc4e7f72e710bebed4d045a4559e50"
server_key_iv: "e22c3f9a61310a94c1124efc178176ba"
server_key_version: "encryptionkey@0"
I20260923 17:29:12.591481 1786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:29:12.606968 1786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:29:12.610388 1786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:29:12.610599 1786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:29:12.617946 1786 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:34749
I20260923 17:29:12.617956 1853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:34749 every 8 connection(s)
I20260923 17:29:12.618945 1786 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:29:12.621831 1854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:29:12.626653 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 1786
I20260923 17:29:12.626824 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:29:12.627089 12180 external_mini_cluster.cc:1428] Setting key aed6645558cd3a9494fefa6f8e7fb47a
I20260923 17:29:12.627564 1854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b574b94cfef04764ab1c9ba1768dbcb9: Bootstrap starting.
I20260923 17:29:12.630095 1854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b574b94cfef04764ab1c9ba1768dbcb9: Neither blocks nor log segments found. Creating new log.
I20260923 17:29:12.630887 1854 log.cc:824] T 00000000000000000000000000000000 P b574b94cfef04764ab1c9ba1768dbcb9: Log is configured to *not* fsync() on all Append() calls
I20260923 17:29:12.632789 1854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b574b94cfef04764ab1c9ba1768dbcb9: No bootstrap required, opened a new log
I20260923 17:29:12.635944 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:12.628528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46629 (local address 127.11.229.62:34749)
0923 17:29:12.628941 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:12.628957 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:12.628993 (+ 36us) server_negotiation.cc:407] Connection header received
0923 17:29:12.629704 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:12.629723 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:12.630060 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:12.630443 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:12.630848 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:12.631845 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:12.632724 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:12.633315 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:12.633503 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:12.634304 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:12.634316 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:12.634330 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:29:12.634716 (+ 386us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:12.634769 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:29:12.634792 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:12.634901 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:12.635039 (+ 138us) server_negotiation.cc:299] Negotiation successful
0923 17:29:12.635243 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":135,"threads_started":1}
I20260923 17:29:12.636596 1854 raft_consensus.cc:348] T 00000000000000000000000000000000 P b574b94cfef04764ab1c9ba1768dbcb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b574b94cfef04764ab1c9ba1768dbcb9" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34749 } }
I20260923 17:29:12.636866 1854 raft_consensus.cc:374] T 00000000000000000000000000000000 P b574b94cfef04764ab1c9ba1768dbcb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:29:12.636940 1854 raft_consensus.cc:729] T 00000000000000000000000000000000 P b574b94cfef04764ab1c9ba1768dbcb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b574b94cfef04764ab1c9ba1768dbcb9, State: Initialized, Role: FOLLOWER
I20260923 17:29:12.637439 1854 consensus_queue.cc:260] T 00000000000000000000000000000000 P b574b94cfef04764ab1c9ba1768dbcb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b574b94cfef04764ab1c9ba1768dbcb9" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34749 } }
I20260923 17:29:12.637598 1854 raft_consensus.cc:388] T 00000000000000000000000000000000 P b574b94cfef04764ab1c9ba1768dbcb9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:29:12.637686 1854 raft_consensus.cc:482] T 00000000000000000000000000000000 P b574b94cfef04764ab1c9ba1768dbcb9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:29:12.637791 1854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b574b94cfef04764ab1c9ba1768dbcb9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:29:12.638804 12180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:29:12.638804 1854 raft_consensus.cc:504] T 00000000000000000000000000000000 P b574b94cfef04764ab1c9ba1768dbcb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b574b94cfef04764ab1c9ba1768dbcb9" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34749 } }
I20260923 17:29:12.639150 1854 leader_election.cc:302] T 00000000000000000000000000000000 P b574b94cfef04764ab1c9ba1768dbcb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b574b94cfef04764ab1c9ba1768dbcb9; no voters:
I20260923 17:29:12.639484 1854 leader_election.cc:288] T 00000000000000000000000000000000 P b574b94cfef04764ab1c9ba1768dbcb9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:29:12.639592 1859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b574b94cfef04764ab1c9ba1768dbcb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:29:12.639832 1859 raft_consensus.cc:686] T 00000000000000000000000000000000 P b574b94cfef04764ab1c9ba1768dbcb9 [term 1 LEADER]: Becoming Leader. State: Replica: b574b94cfef04764ab1c9ba1768dbcb9, State: Running, Role: LEADER
I20260923 17:29:12.640177 1859 consensus_queue.cc:237] T 00000000000000000000000000000000 P b574b94cfef04764ab1c9ba1768dbcb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b574b94cfef04764ab1c9ba1768dbcb9" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34749 } }
I20260923 17:29:12.640825 1854 sys_catalog.cc:564] T 00000000000000000000000000000000 P b574b94cfef04764ab1c9ba1768dbcb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:29:12.641639 1860 sys_catalog.cc:455] T 00000000000000000000000000000000 P b574b94cfef04764ab1c9ba1768dbcb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b574b94cfef04764ab1c9ba1768dbcb9. Latest consensus state: current_term: 1 leader_uuid: "b574b94cfef04764ab1c9ba1768dbcb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b574b94cfef04764ab1c9ba1768dbcb9" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34749 } } }
I20260923 17:29:12.641788 1860 sys_catalog.cc:458] T 00000000000000000000000000000000 P b574b94cfef04764ab1c9ba1768dbcb9 [sys.catalog]: This master's current role is: LEADER
I20260923 17:29:12.642473 1861 sys_catalog.cc:455] T 00000000000000000000000000000000 P b574b94cfef04764ab1c9ba1768dbcb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b574b94cfef04764ab1c9ba1768dbcb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b574b94cfef04764ab1c9ba1768dbcb9" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34749 } } }
I20260923 17:29:12.642558 1861 sys_catalog.cc:458] T 00000000000000000000000000000000 P b574b94cfef04764ab1c9ba1768dbcb9 [sys.catalog]: This master's current role is: LEADER
I20260923 17:29:12.643445 1873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:29:12.645779 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:12.640981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53331 (local address 127.11.229.62:34749)
0923 17:29:12.641089 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:12.641096 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:12.641483 (+ 387us) server_negotiation.cc:407] Connection header received
0923 17:29:12.641670 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:12.641674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:12.641722 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:12.641776 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:12.642242 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:12.643016 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:12.644151 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:12.644388 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:12.644468 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:12.645347 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:12.645353 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:12.645356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:29:12.645368 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:12.645396 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:29:12.645410 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:12.645490 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:12.645591 (+ 101us) server_negotiation.cc:299] Negotiation successful
0923 17:29:12.645649 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260923 17:29:12.647054 1873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:29:12.652671 1873 catalog_manager.cc:1347] Generated new cluster ID: 8fcb273f38d54fa1af9b179b672f8f7b
I20260923 17:29:12.652737 1873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:29:12.698277 1873 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:29:12.699555 1873 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:29:12.705754 1873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b574b94cfef04764ab1c9ba1768dbcb9: Generated new TSK 0
I20260923 17:29:12.706534 1873 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:29:12.723057 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 1786
2026-09-23T17:29:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1490 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:29:12 dist-test-slave-ptl9 krb5kdc[1885](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:29:12 dist-test-slave-ptl9 krb5kdc[1885](info): set up 2 sockets
Sep 23 17:29:12 dist-test-slave-ptl9 krb5kdc[1885](info): commencing operation
krb5kdc: starting...
W20260923 17:29:13.761873 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:29:13 dist-test-slave-ptl9 krb5kdc[1885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:29:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:29:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.229.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
I20260923 17:29:13.888907 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:37495
--webserver_interface=127.11.229.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:44873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:37495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:29:14.003182 1898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:14.003557 1898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:14.003651 1898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:29:14.008095 1898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:29:14.008217 1898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:14.008257 1898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:14.008301 1898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:29:14.008361 1898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:29:14.014362 1898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:44873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:37495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:37495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:29:14.015586 1898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:14.016637 1898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:14.025542 1903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:14.025539 1906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:14.025535 1904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:14.025635 1898 server_base.cc:1034] running on GCE node
I20260923 17:29:14.026309 1898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:14.027427 1898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:14.028620 1898 hybrid_clock.cc:648] HybridClock initialized: now 1790184554028606 us; error 27 us; skew 500 ppm
Sep 23 17:29:14 dist-test-slave-ptl9 krb5kdc[1885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184554, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:29:14.031903 1898 init.cc:377] Logged in from keytab as kudu/127.11.229.62@KRBTEST.COM (short username kudu)
I20260923 17:29:14.032866 1898 webserver.cc:460] Webserver started at http://127.11.229.62:34841/ using document root <none> and password file <none>
I20260923 17:29:14.033528 1898 fs_manager.cc:359] Metadata directory not provided
I20260923 17:29:14.033620 1898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:29:14.033861 1898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:29:14.035971 1898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "52b507042f134869849d3804ebaf2079"
format_stamp: "Formatted at 2026-09-23 17:29:14 on dist-test-slave-ptl9"
server_key: "97478d2f9eecfd1fee5095ff53dc2fb0"
server_key_iv: "f22746867e4fcb0c3365382507f0301f"
server_key_version: "encryptionkey@0"
I20260923 17:29:14.036552 1898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "52b507042f134869849d3804ebaf2079"
format_stamp: "Formatted at 2026-09-23 17:29:14 on dist-test-slave-ptl9"
server_key: "97478d2f9eecfd1fee5095ff53dc2fb0"
server_key_iv: "f22746867e4fcb0c3365382507f0301f"
server_key_version: "encryptionkey@0"
I20260923 17:29:14.040318 1898 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260923 17:29:14.042776 1913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:14.043998 1898 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:29:14.044138 1898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "52b507042f134869849d3804ebaf2079"
format_stamp: "Formatted at 2026-09-23 17:29:14 on dist-test-slave-ptl9"
server_key: "97478d2f9eecfd1fee5095ff53dc2fb0"
server_key_iv: "f22746867e4fcb0c3365382507f0301f"
server_key_version: "encryptionkey@0"
I20260923 17:29:14.044252 1898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:29:14.048902 1898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:29:14.052191 1898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:29:14.052392 1898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:29:14.060036 1898 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:37495
I20260923 17:29:14.060056 1965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:37495 every 8 connection(s)
I20260923 17:29:14.061028 1898 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:29:14.063865 1966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:29:14.065583 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 1898
I20260923 17:29:14.065745 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:29:14.066025 12180 external_mini_cluster.cc:1428] Setting key bd6da705b4c6d735c47abfd579f6059a
I20260923 17:29:14.070524 1966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52b507042f134869849d3804ebaf2079: Bootstrap starting.
I20260923 17:29:14.071472 1969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:14.067509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42681 (local address 127.11.229.62:37495)
0923 17:29:14.068040 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:14.068055 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:14.068093 (+ 38us) server_negotiation.cc:407] Connection header received
0923 17:29:14.068756 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:14.068774 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:14.069070 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:14.069463 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:14.070263 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:14.070268 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:14.070279 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:29:14.070289 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:14.070343 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:29:14.070362 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:14.070517 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:14.070589 (+ 72us) server_negotiation.cc:299] Negotiation successful
0923 17:29:14.070757 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":144,"threads_started":1}
I20260923 17:29:14.073028 1966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52b507042f134869849d3804ebaf2079: Neither blocks nor log segments found. Creating new log.
I20260923 17:29:14.073956 1966 log.cc:824] T 00000000000000000000000000000000 P 52b507042f134869849d3804ebaf2079: Log is configured to *not* fsync() on all Append() calls
I20260923 17:29:14.074349 12180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:29:14.076329 1966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52b507042f134869849d3804ebaf2079: No bootstrap required, opened a new log
I20260923 17:29:14.078265 1969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:14.076602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47949 (local address 127.11.229.62:37495)
0923 17:29:14.076736 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:14.076740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:14.076980 (+ 240us) server_negotiation.cc:407] Connection header received
0923 17:29:14.077098 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:14.077100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:14.077129 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:14.077192 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:14.077875 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:14.077879 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:14.077880 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:29:14.077891 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:14.077916 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:29:14.077926 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:14.077993 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:14.078095 (+ 102us) server_negotiation.cc:299] Negotiation successful
0923 17:29:14.078125 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 17:29:14.080750 1966 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52b507042f134869849d3804ebaf2079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52b507042f134869849d3804ebaf2079" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 37495 } }
I20260923 17:29:14.081007 1966 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52b507042f134869849d3804ebaf2079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:29:14.081067 1966 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52b507042f134869849d3804ebaf2079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52b507042f134869849d3804ebaf2079, State: Initialized, Role: FOLLOWER
I20260923 17:29:14.081568 1966 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52b507042f134869849d3804ebaf2079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52b507042f134869849d3804ebaf2079" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 37495 } }
I20260923 17:29:14.081727 1966 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52b507042f134869849d3804ebaf2079 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:29:14.081812 1966 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52b507042f134869849d3804ebaf2079 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:29:14.081910 1966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52b507042f134869849d3804ebaf2079 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:29:14.082845 1966 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52b507042f134869849d3804ebaf2079 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52b507042f134869849d3804ebaf2079" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 37495 } }
I20260923 17:29:14.083196 1966 leader_election.cc:302] T 00000000000000000000000000000000 P 52b507042f134869849d3804ebaf2079 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52b507042f134869849d3804ebaf2079; no voters:
I20260923 17:29:14.083505 1966 leader_election.cc:288] T 00000000000000000000000000000000 P 52b507042f134869849d3804ebaf2079 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:29:14.083617 1976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52b507042f134869849d3804ebaf2079 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:29:14.083860 1976 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52b507042f134869849d3804ebaf2079 [term 1 LEADER]: Becoming Leader. State: Replica: 52b507042f134869849d3804ebaf2079, State: Running, Role: LEADER
I20260923 17:29:14.084185 1976 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52b507042f134869849d3804ebaf2079 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52b507042f134869849d3804ebaf2079" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 37495 } }
I20260923 17:29:14.084733 1966 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52b507042f134869849d3804ebaf2079 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:29:14.085942 1978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52b507042f134869849d3804ebaf2079 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52b507042f134869849d3804ebaf2079. Latest consensus state: current_term: 1 leader_uuid: "52b507042f134869849d3804ebaf2079" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52b507042f134869849d3804ebaf2079" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 37495 } } }
I20260923 17:29:14.085933 1977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52b507042f134869849d3804ebaf2079 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52b507042f134869849d3804ebaf2079" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52b507042f134869849d3804ebaf2079" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 37495 } } }
I20260923 17:29:14.086104 1978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52b507042f134869849d3804ebaf2079 [sys.catalog]: This master's current role is: LEADER
I20260923 17:29:14.086119 1977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52b507042f134869849d3804ebaf2079 [sys.catalog]: This master's current role is: LEADER
I20260923 17:29:14.086517 1985 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:29:14.089473 1985 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:29:14.094823 1985 catalog_manager.cc:1347] Generated new cluster ID: 031a810a85084800ab01c44d94833f49
I20260923 17:29:14.094901 1985 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:29:14.120301 1985 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:29:14.120718 1985 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:29:14.128000 1985 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52b507042f134869849d3804ebaf2079: Generated new TSK 0
I20260923 17:29:14.128832 1985 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:29:14.134582 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 1898
2026-09-23T17:29:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1411 ms)
[----------] 18 tests from AuthTokenIssuingTest (25640 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:29:14 dist-test-slave-ptl9 krb5kdc[1997](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:29:14 dist-test-slave-ptl9 krb5kdc[1997](info): set up 2 sockets
Sep 23 17:29:14 dist-test-slave-ptl9 krb5kdc[1997](info): commencing operation
krb5kdc: starting...
W20260923 17:29:15.178339 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:29:15 dist-test-slave-ptl9 krb5kdc[1997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:29:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:29:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.229.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
I20260923 17:29:15.304922 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:33861
--webserver_interface=127.11.229.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:43255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:33861,127.11.229.61:34135,127.11.229.60:42963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:29:15.418664 2010 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 17:29:15.419014 2010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:15.419078 2010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:15.419157 2010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:29:15.423233 2010 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:29:15.423316 2010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:15.423352 2010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:15.423408 2010 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:29:15.423444 2010 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:29:15.428766 2010 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:43255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:33861,127.11.229.61:34135,127.11.229.60:42963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:33861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:29:15.429952 2010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:15.430994 2010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:15.439635 2016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:15.439635 2015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:15.439829 2010 server_base.cc:1034] running on GCE node
W20260923 17:29:15.439639 2018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:15.440557 2010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:15.441735 2010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:15.442915 2010 hybrid_clock.cc:648] HybridClock initialized: now 1790184555442900 us; error 30 us; skew 500 ppm
Sep 23 17:29:15 dist-test-slave-ptl9 krb5kdc[1997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184555, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:29:15.450026 2010 init.cc:377] Logged in from keytab as kudu/127.11.229.62@KRBTEST.COM (short username kudu)
I20260923 17:29:15.451083 2010 webserver.cc:460] Webserver started at http://127.11.229.62:40689/ using document root <none> and password file <none>
I20260923 17:29:15.451689 2010 fs_manager.cc:359] Metadata directory not provided
I20260923 17:29:15.451786 2010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:29:15.452036 2010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:29:15.454281 2010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "09d1eec189fd4898b279d8cdd45a737c"
format_stamp: "Formatted at 2026-09-23 17:29:15 on dist-test-slave-ptl9"
server_key: "a6c9c62da24c2d4f644d56bce2808f68"
server_key_iv: "1e656bf50e911136cc3fbc68e0d9ec93"
server_key_version: "encryptionkey@0"
I20260923 17:29:15.454875 2010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "09d1eec189fd4898b279d8cdd45a737c"
format_stamp: "Formatted at 2026-09-23 17:29:15 on dist-test-slave-ptl9"
server_key: "a6c9c62da24c2d4f644d56bce2808f68"
server_key_iv: "1e656bf50e911136cc3fbc68e0d9ec93"
server_key_version: "encryptionkey@0"
I20260923 17:29:15.458729 2010 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:29:15.461362 2025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:15.462643 2010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:29:15.462791 2010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "09d1eec189fd4898b279d8cdd45a737c"
format_stamp: "Formatted at 2026-09-23 17:29:15 on dist-test-slave-ptl9"
server_key: "a6c9c62da24c2d4f644d56bce2808f68"
server_key_iv: "1e656bf50e911136cc3fbc68e0d9ec93"
server_key_version: "encryptionkey@0"
I20260923 17:29:15.462919 2010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:29:15.475829 2010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:29:15.478724 2010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:29:15.478940 2010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:29:15.486013 2010 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:33861
I20260923 17:29:15.486315 2077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:33861 every 8 connection(s)
I20260923 17:29:15.487039 2010 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:29:15.489946 2078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:29:15.491267 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 2010
I20260923 17:29:15.491429 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:29:15.491757 12180 external_mini_cluster.cc:1428] Setting key 8ce3ec07886607654e677c96c8aaa542
I20260923 17:29:15.494973 2078 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.229.62" port: 33861 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.11.229.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.61@KRBTEST.COM" created.
Sep 23 17:29:15 dist-test-slave-ptl9 krb5kdc[1997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184555, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:29:15.511478 2081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.497563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51556 (local address 127.11.229.62:33861)
0923 17:29:15.498013 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:15.498034 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:15.498320 (+ 286us) server_negotiation.cc:407] Connection header received
0923 17:29:15.499409 (+ 1089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:15.499414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:15.499602 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:15.499796 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:15.500688 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.501710 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.502741 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.503220 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.503411 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:15.506251 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:15.506257 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:15.506269 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:15.506290 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:15.508091 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:15.508597 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:15.508605 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:15.508612 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:15.508713 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:15.509045 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:15.509048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:15.509050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:15.509543 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:15.509691 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:15.510059 (+ 368us) server_negotiation.cc:299] Negotiation successful
0923 17:29:15.510434 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":158,"server-negotiator.queue_time_us":294,"thread_start_us":161,"threads_started":1}
I20260923 17:29:15.511478 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.497564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:33861 (local address 127.0.0.1:51556)
0923 17:29:15.498171 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:15.498207 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:15.499088 (+ 881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:15.499794 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:15.499799 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:15.500272 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:15.500533 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:15.500547 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.501836 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.501840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:15.502582 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:15.502592 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.503463 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.503467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:15.503579 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:15.504246 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:15.504263 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:15.506084 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:29:15.508227 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:15.508238 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:15.508248 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:15.508480 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:15.508820 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:15.508825 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:15.508827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:15.508943 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:15.509659 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:29:15.509666 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:29:15.509817 (+ 151us) client_negotiation.cc:769] Sending connection context
0923 17:29:15.509995 (+ 178us) client_negotiation.cc:240] Negotiation successful
0923 17:29:15.510385 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":2,"spinlock_wait_cycles":21888,"thread_start_us":144,"threads_started":1}
I20260923 17:29:15.514196 2078 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.229.61" port: 34135 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:29:15.514994 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.514710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:34135 (local address 127.0.0.1:33670)
0923 17:29:15.514850 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:15.514925 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.61:34135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260923 17:29:15.515255 2029 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.229.61:34135: connect: Connection refused (error 111)
I20260923 17:29:15.516191 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.515967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:34135 (local address 127.0.0.1:33686)
0923 17:29:15.516096 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:15.516140 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.61:34135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260923 17:29:15.516403 2078 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.229.61:34135: Network error: Client connection negotiation failed: client connection to 127.11.229.61:34135: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.61.keytab.
Entry for principal kudu/127.11.229.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.61.keytab.
I20260923 17:29:15.520220 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.61:34135
--webserver_interface=127.11.229.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.61.keytab
--principal=kudu/127.11.229.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:43255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:33861,127.11.229.61:34135,127.11.229.60:42963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:29:15.565644 2078 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.229.61" port: 34135 } attempt: 1
I20260923 17:29:15.566699 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.566290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:34135 (local address 127.0.0.1:33694)
0923 17:29:15.566520 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:15.566588 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.61:34135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260923 17:29:15.567516 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.567298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:34135 (local address 127.0.0.1:33702)
0923 17:29:15.567409 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:15.567456 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.61:34135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260923 17:29:15.567713 2078 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.229.61:34135: Network error: Client connection negotiation failed: client connection to 127.11.229.61:34135: connect: Connection refused (error 111)
W20260923 17:29:15.632056 2085 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 17:29:15.632423 2085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:15.632478 2085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:15.632587 2085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260923 17:29:15.635885 2078 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.229.61" port: 34135 } attempt: 2
I20260923 17:29:15.636722 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.636412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:34135 (local address 127.0.0.1:33718)
0923 17:29:15.636575 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:15.636643 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.61:34135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260923 17:29:15.636806 2085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:29:15.636900 2085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:15.636930 2085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:15.636969 2085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:29:15.636993 2085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:29:15.637616 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.637379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:34135 (local address 127.0.0.1:33722)
0923 17:29:15.637509 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:15.637556 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.61:34135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260923 17:29:15.637751 2078 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.229.61:34135: Network error: Client connection negotiation failed: client connection to 127.11.229.61:34135: connect: Connection refused (error 111)
I20260923 17:29:15.642665 2085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:43255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:33861,127.11.229.61:34135,127.11.229.60:42963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.61.keytab
--principal=kudu/127.11.229.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.61:34135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.229.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:29:15.643806 2085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:15.644860 2085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:15.653599 2093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:15.653685 2085 server_base.cc:1034] running on GCE node
W20260923 17:29:15.653622 2090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:15.653607 2091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:15.654455 2085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:15.655606 2085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:15.656771 2085 hybrid_clock.cc:648] HybridClock initialized: now 1790184555656757 us; error 30 us; skew 500 ppm
Sep 23 17:29:15 dist-test-slave-ptl9 krb5kdc[1997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184555, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:29:15.659585 2085 init.cc:377] Logged in from keytab as kudu/127.11.229.61@KRBTEST.COM (short username kudu)
I20260923 17:29:15.660483 2085 webserver.cc:460] Webserver started at http://127.11.229.61:45363/ using document root <none> and password file <none>
I20260923 17:29:15.661064 2085 fs_manager.cc:359] Metadata directory not provided
I20260923 17:29:15.661158 2085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:29:15.661460 2085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:29:15.663604 2085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-1/data/instance:
uuid: "25df9595488b4c52b19ac5d6ca719818"
format_stamp: "Formatted at 2026-09-23 17:29:15 on dist-test-slave-ptl9"
server_key: "e1600a402823a34956eed71b6c134228"
server_key_iv: "cac8eefbb94af5c6fb7c6b5b0791a77b"
server_key_version: "encryptionkey@0"
I20260923 17:29:15.664212 2085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-1/wal/instance:
uuid: "25df9595488b4c52b19ac5d6ca719818"
format_stamp: "Formatted at 2026-09-23 17:29:15 on dist-test-slave-ptl9"
server_key: "e1600a402823a34956eed71b6c134228"
server_key_iv: "cac8eefbb94af5c6fb7c6b5b0791a77b"
server_key_version: "encryptionkey@0"
I20260923 17:29:15.668218 2085 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260923 17:29:15.670786 2100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:15.672052 2085 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:29:15.672200 2085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-1/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-1/wal
uuid: "25df9595488b4c52b19ac5d6ca719818"
format_stamp: "Formatted at 2026-09-23 17:29:15 on dist-test-slave-ptl9"
server_key: "e1600a402823a34956eed71b6c134228"
server_key_iv: "cac8eefbb94af5c6fb7c6b5b0791a77b"
server_key_version: "encryptionkey@0"
I20260923 17:29:15.672319 2085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:29:15.695063 2085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:29:15.698419 2085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:29:15.698638 2085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:29:15.706087 2085 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.61:34135
I20260923 17:29:15.706405 2152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.61:34135 every 8 connection(s)
I20260923 17:29:15.707198 2085 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-1/data/info.pb
I20260923 17:29:15.710151 2153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:29:15.714160 2153 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.229.62" port: 33861 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:29:15.716655 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 2085
I20260923 17:29:15.716806 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-1/wal/instance
I20260923 17:29:15.717125 12180 external_mini_cluster.cc:1428] Setting key cb4a206a020989637cc4fd3146396802
Sep 23 17:29:15 dist-test-slave-ptl9 krb5kdc[1997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184555, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.61@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.229.60@KRBTEST.COM; defaulting to no policy
I20260923 17:29:15.728421 2081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.716087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51568 (local address 127.11.229.62:33861)
0923 17:29:15.716233 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:15.716238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:15.717140 (+ 902us) server_negotiation.cc:407] Connection header received
0923 17:29:15.718128 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:15.718135 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:15.718207 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:15.718318 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:15.719467 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.720172 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.721450 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.721693 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.721814 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:15.724657 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:15.724660 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:15.724662 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:15.724688 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:15.726231 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:15.726789 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:15.726802 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:15.726804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:15.726889 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:15.727248 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:15.727252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:15.727254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:15.727448 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:15.727564 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:15.728108 (+ 544us) server_negotiation.cc:299] Negotiation successful
0923 17:29:15.728238 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.60@KRBTEST.COM" created.
I20260923 17:29:15.728901 2078 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.229.61" port: 34135 } attempt: 3
I20260923 17:29:15.729590 2155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.716487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:33861 (local address 127.0.0.1:51568)
0923 17:29:15.716967 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:15.717020 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:15.717916 (+ 896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:15.718477 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:15.718484 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:15.718823 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:15.719304 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:15.719319 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.720322 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.720326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:15.721216 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:15.721227 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.721855 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.721859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:15.722093 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:15.722743 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:15.722769 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:15.724460 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:29:15.726353 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:15.726359 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:15.726375 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:15.726651 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:15.726992 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:15.726995 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:15.726998 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:15.727148 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:15.727564 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:29:15.727572 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:29:15.727844 (+ 272us) client_negotiation.cc:769] Sending connection context
0923 17:29:15.728051 (+ 207us) client_negotiation.cc:240] Negotiation successful
0923 17:29:15.728385 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":138,"threads_started":1}
I20260923 17:29:15.731045 2153 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.229.61" port: 34135 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:29:15 dist-test-slave-ptl9 krb5kdc[1997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184555, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.61@KRBTEST.COM for kudu/127.11.229.61@KRBTEST.COM
Sep 23 17:29:15 dist-test-slave-ptl9 krb5kdc[1997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184555, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for kudu/127.11.229.61@KRBTEST.COM
I20260923 17:29:15.741784 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.729370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:34135 (local address 127.0.0.1:33730)
0923 17:29:15.729947 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:15.729960 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:15.730058 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:15.730740 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:15.730745 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:15.730945 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:15.731080 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:15.731089 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.732360 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.732365 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:15.733761 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:15.733772 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.734858 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.734864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:15.735011 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:15.735629 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:15.735644 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:15.737367 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:29:15.739799 (+ 2432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:15.739802 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:15.739804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:15.740100 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:15.740419 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:15.740425 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:15.740427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:15.740564 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:15.741208 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:29:15.741212 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:29:15.741374 (+ 162us) client_negotiation.cc:769] Sending connection context
0923 17:29:15.741474 (+ 100us) client_negotiation.cc:240] Negotiation successful
0923 17:29:15.741611 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479}
I20260923 17:29:15.741784 2155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.731564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:34135 (local address 127.0.0.1:33734)
0923 17:29:15.731733 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:15.731747 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:15.731822 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:15.732085 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:15.732088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:15.732270 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:15.732407 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:15.732416 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.733570 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.733573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:15.734287 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:15.734297 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.734840 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.734843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:15.734932 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:15.735439 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:15.735458 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:15.736748 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:29:15.739940 (+ 3192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:15.739946 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:15.739949 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:15.740178 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:15.740484 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:15.740486 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:15.740488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:15.740592 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:15.741208 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:29:15.741212 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:29:15.741343 (+ 131us) client_negotiation.cc:769] Sending connection context
0923 17:29:15.741479 (+ 136us) client_negotiation.cc:240] Negotiation successful
0923 17:29:15.741611 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":40}
I20260923 17:29:15.741959 2159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.731583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33734 (local address 127.11.229.61:34135)
0923 17:29:15.731896 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:15.731908 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:15.731918 (+ 10us) server_negotiation.cc:407] Connection header received
0923 17:29:15.731945 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:15.731948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:15.731986 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:15.732074 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:15.732532 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.733444 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.734406 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.734725 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.734840 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:15.736922 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:15.736936 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:15.736955 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:15.736983 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:15.739810 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:15.740276 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:15.740279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:15.740280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:15.740354 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:15.740708 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:15.740714 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:15.740717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:15.741083 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:15.741225 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:15.741597 (+ 372us) server_negotiation.cc:299] Negotiation successful
0923 17:29:15.741720 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":230,"spinlock_wait_cycles":1920,"thread_start_us":109,"threads_started":1}
I20260923 17:29:15.742722 2157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.729560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33730 (local address 127.11.229.61:34135)
0923 17:29:15.730301 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:15.730314 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:15.730339 (+ 25us) server_negotiation.cc:407] Connection header received
0923 17:29:15.730389 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:15.730398 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:15.730601 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:15.730737 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:15.731229 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.732210 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.733891 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.734694 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.735247 (+ 553us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:15.737541 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:15.737548 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:15.737552 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:15.737584 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:15.739684 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:15.740222 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:15.740231 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:15.740233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:15.740323 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:15.740668 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:15.740674 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:15.740675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:15.741083 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:15.741225 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:15.742467 (+ 1242us) server_negotiation.cc:299] Negotiation successful
0923 17:29:15.742597 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":115,"server-negotiator.queue_time_us":639,"spinlock_wait_cycles":1408,"thread_start_us":139,"threads_started":1}
I20260923 17:29:15.743819 2078 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.229.60" port: 42963 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:29:15.743953 2153 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.229.60" port: 42963 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:29:15.744655 2155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.744447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:42963 (local address 127.0.0.1:50492)
0923 17:29:15.744542 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:15.744606 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.60:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260923 17:29:15.744668 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.744357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:42963 (local address 127.0.0.1:50488)
0923 17:29:15.744490 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:15.744548 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.60:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260923 17:29:15.744820 2102 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.229.60:42963: connect: Connection refused (error 111)
I20260923 17:29:15.745399 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.745135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:42963 (local address 127.0.0.1:50494)
0923 17:29:15.745298 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:15.745344 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.60:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260923 17:29:15.745551 2078 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.229.60:42963: Network error: Client connection negotiation failed: client connection to 127.11.229.60:42963: connect: Connection refused (error 111)
I20260923 17:29:15.745858 2155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.745657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:42963 (local address 127.0.0.1:50510)
0923 17:29:15.745769 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:15.745813 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.60:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260923 17:29:15.745993 2153 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.229.60:42963: Network error: Client connection negotiation failed: client connection to 127.11.229.60:42963: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.60.keytab.
Entry for principal kudu/127.11.229.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.60.keytab.
I20260923 17:29:15.748274 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.60:42963
--webserver_interface=127.11.229.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.60.keytab
--principal=kudu/127.11.229.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:43255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:33861,127.11.229.61:34135,127.11.229.60:42963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:29:15.776708 2078 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.229.60" port: 42963 } attempt: 1
I20260923 17:29:15.777631 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.777328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:42963 (local address 127.0.0.1:50522)
0923 17:29:15.777486 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:15.777551 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.60:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260923 17:29:15.778321 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.778100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:42963 (local address 127.0.0.1:50528)
0923 17:29:15.778218 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:15.778269 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.60:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260923 17:29:15.778470 2078 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.229.60:42963: Network error: Client connection negotiation failed: client connection to 127.11.229.60:42963: connect: Connection refused (error 111)
I20260923 17:29:15.795297 2153 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.229.60" port: 42963 } attempt: 1
I20260923 17:29:15.796188 2155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.795844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:42963 (local address 127.0.0.1:50536)
0923 17:29:15.796021 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:15.796093 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.60:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260923 17:29:15.796995 2155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.796754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:42963 (local address 127.0.0.1:50548)
0923 17:29:15.796874 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:15.796929 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.60:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260923 17:29:15.797173 2153 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.229.60:42963: Network error: Client connection negotiation failed: client connection to 127.11.229.60:42963: connect: Connection refused (error 111)
I20260923 17:29:15.853619 2078 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.229.60" port: 42963 } attempt: 2
I20260923 17:29:15.854563 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.854201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:42963 (local address 127.0.0.1:50564)
0923 17:29:15.854397 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:15.854469 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.60:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260923 17:29:15.855325 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.855084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:42963 (local address 127.0.0.1:50570)
0923 17:29:15.855203 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:15.855264 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.60:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 17:29:15.855501 2078 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.229.60:42963: Network error: Client connection negotiation failed: client connection to 127.11.229.60:42963: connect: Connection refused (error 111)
W20260923 17:29:15.859285 2161 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 17:29:15.859602 2161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:15.859663 2161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:15.859738 2161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:29:15.863788 2161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:29:15.863869 2161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:15.863906 2161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:15.863962 2161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:29:15.863999 2161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:29:15.865322 2153 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.229.60" port: 42963 } attempt: 2
I20260923 17:29:15.866124 2155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.865794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:42963 (local address 127.0.0.1:50574)
0923 17:29:15.865931 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:15.866002 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.60:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260923 17:29:15.866904 2155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.866690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:42963 (local address 127.0.0.1:50580)
0923 17:29:15.866796 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:15.866843 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.60:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260923 17:29:15.867054 2153 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.229.60:42963: Network error: Client connection negotiation failed: client connection to 127.11.229.60:42963: connect: Connection refused (error 111)
I20260923 17:29:15.869812 2161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:43255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:33861,127.11.229.61:34135,127.11.229.60:42963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.60.keytab
--principal=kudu/127.11.229.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.60:42963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.229.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:29:15.870891 2161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:15.871922 2161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:15.880538 2167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:15.880544 2169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:15.880545 2166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:15.880959 2161 server_base.cc:1034] running on GCE node
I20260923 17:29:15.881577 2161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:15.882689 2161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:15.883849 2161 hybrid_clock.cc:648] HybridClock initialized: now 1790184555883834 us; error 29 us; skew 500 ppm
Sep 23 17:29:15 dist-test-slave-ptl9 krb5kdc[1997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184555, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:29:15.886675 2161 init.cc:377] Logged in from keytab as kudu/127.11.229.60@KRBTEST.COM (short username kudu)
I20260923 17:29:15.887609 2161 webserver.cc:460] Webserver started at http://127.11.229.60:38381/ using document root <none> and password file <none>
I20260923 17:29:15.888170 2161 fs_manager.cc:359] Metadata directory not provided
I20260923 17:29:15.888260 2161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:29:15.888530 2161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:29:15.890663 2161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-2/data/instance:
uuid: "384fcf05ec7b40e08c8317633fecb9be"
format_stamp: "Formatted at 2026-09-23 17:29:15 on dist-test-slave-ptl9"
server_key: "fd03fdae17e212ab8f5bd9d77fac6cc4"
server_key_iv: "38980ca5c8fb8b6495b88bcf75906cda"
server_key_version: "encryptionkey@0"
I20260923 17:29:15.891232 2161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-2/wal/instance:
uuid: "384fcf05ec7b40e08c8317633fecb9be"
format_stamp: "Formatted at 2026-09-23 17:29:15 on dist-test-slave-ptl9"
server_key: "fd03fdae17e212ab8f5bd9d77fac6cc4"
server_key_iv: "38980ca5c8fb8b6495b88bcf75906cda"
server_key_version: "encryptionkey@0"
I20260923 17:29:15.895021 2161 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:29:15.897485 2176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:15.898739 2161 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 17:29:15.898901 2161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-2/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-2/wal
uuid: "384fcf05ec7b40e08c8317633fecb9be"
format_stamp: "Formatted at 2026-09-23 17:29:15 on dist-test-slave-ptl9"
server_key: "fd03fdae17e212ab8f5bd9d77fac6cc4"
server_key_iv: "38980ca5c8fb8b6495b88bcf75906cda"
server_key_version: "encryptionkey@0"
I20260923 17:29:15.899022 2161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:29:15.919490 2161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:29:15.922428 2161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:29:15.922640 2161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:29:15.929860 2161 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.60:42963
I20260923 17:29:15.930181 2228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.60:42963 every 8 connection(s)
I20260923 17:29:15.930871 2161 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-2/data/info.pb
I20260923 17:29:15.933768 2229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:29:15.934281 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 2161
I20260923 17:29:15.934446 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184271597484-12180-0/minicluster-data/master-2/wal/instance
I20260923 17:29:15.934734 12180 external_mini_cluster.cc:1428] Setting key d729d7843dc83881a571f3fd558646ee
I20260923 17:29:15.938832 2229 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.229.62" port: 33861 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:29:15 dist-test-slave-ptl9 krb5kdc[1997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.60@KRBTEST.COM
Sep 23 17:29:15 dist-test-slave-ptl9 krb5kdc[1997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184555, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.60@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:29:15.949766 2232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.936036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50582 (local address 127.11.229.60:42963)
0923 17:29:15.936459 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:15.936483 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:15.936519 (+ 36us) server_negotiation.cc:407] Connection header received
0923 17:29:15.937189 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:15.937205 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:15.937608 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:15.937897 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:15.938828 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.939888 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.940928 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.941630 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.941804 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:15.944521 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:15.944527 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:15.944536 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:15.944563 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:15.946522 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:15.946961 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:15.946966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:15.946970 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:15.947150 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:15.947489 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:15.947492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:15.947493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:15.948047 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:15.948254 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:15.948613 (+ 359us) server_negotiation.cc:299] Negotiation successful
0923 17:29:15.948906 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"spinlock_wait_cycles":28928,"thread_start_us":128,"threads_started":1}
I20260923 17:29:15.950232 2233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.940559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:33861 (local address 127.0.0.1:51570)
0923 17:29:15.940975 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:15.941024 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:15.941146 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:15.941458 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:15.941463 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:15.941785 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:15.941930 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:15.941936 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.942680 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.942685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:15.943347 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:15.943371 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.943884 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.943892 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:15.944011 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:15.944588 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:15.944601 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:15.946432 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:29:15.948373 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:15.948382 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:15.948389 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:15.948708 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:15.949056 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:15.949059 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:15.949061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:15.949142 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:15.949604 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:29:15.949611 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:29:15.949720 (+ 109us) client_negotiation.cc:769] Sending connection context
0923 17:29:15.949870 (+ 150us) client_negotiation.cc:240] Negotiation successful
0923 17:29:15.950058 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":5,"thread_start_us":180,"threads_started":1}
I20260923 17:29:15.950431 2081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.940634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51570 (local address 127.11.229.62:33861)
0923 17:29:15.940783 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:15.940788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:15.941115 (+ 327us) server_negotiation.cc:407] Connection header received
0923 17:29:15.941240 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:15.941243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:15.941315 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:15.941401 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:15.942067 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.942557 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.943508 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.943723 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.943805 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:15.946585 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:15.946591 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:15.946598 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:15.946630 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:15.948254 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:15.948859 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:15.948864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:15.948865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:15.948950 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:15.949243 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:15.949246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:15.949248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:15.949497 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:15.949605 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:15.950151 (+ 546us) server_negotiation.cc:299] Negotiation successful
0923 17:29:15.950286 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 17:29:15.951573 2229 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.229.61" port: 34135 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:29:15.954679 2078 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.229.60" port: 42963 } attempt: 3
Sep 23 17:29:15 dist-test-slave-ptl9 krb5kdc[1997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184555, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.60@KRBTEST.COM for kudu/127.11.229.61@KRBTEST.COM
Sep 23 17:29:15 dist-test-slave-ptl9 krb5kdc[1997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:29:15.958201 2153 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.229.60" port: 42963 } attempt: 3
Sep 23 17:29:15 dist-test-slave-ptl9 krb5kdc[1997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184555, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for kudu/127.11.229.60@KRBTEST.COM
I20260923 17:29:15.961768 2157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.952455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33736 (local address 127.11.229.61:34135)
0923 17:29:15.952603 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:15.952607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:15.952627 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:29:15.952664 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:15.952666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:15.952704 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:15.952793 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:15.953187 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.953773 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.954544 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.954911 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.955012 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:15.957788 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:15.957792 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:15.957795 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:15.957826 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:15.959788 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:15.960286 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:15.960290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:15.960291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:15.960383 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:15.960691 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:15.960694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:15.960695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:15.960929 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:15.961002 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:15.961426 (+ 424us) server_negotiation.cc:299] Negotiation successful
0923 17:29:15.961556 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 17:29:15.962255 2081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.952289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51580 (local address 127.11.229.62:33861)
0923 17:29:15.952420 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:15.952425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:15.952438 (+ 13us) server_negotiation.cc:407] Connection header received
0923 17:29:15.952475 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:15.952480 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:15.952524 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:15.952625 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:15.953235 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.953770 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.954642 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.954930 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.955040 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:15.958288 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:15.958292 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:15.958294 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:15.958321 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:15.959961 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:15.960463 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:15.960469 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:15.960473 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:15.960550 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:15.960897 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:15.960903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:15.960905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:15.961149 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:15.961398 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:15.961774 (+ 376us) server_negotiation.cc:299] Negotiation successful
0923 17:29:15.962125 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 17:29:15.962312 2233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.952179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:34135 (local address 127.0.0.1:33736)
0923 17:29:15.952335 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:15.952392 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:15.952465 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:15.952807 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:15.952810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:15.952978 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:15.953067 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:15.953072 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.953902 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.953905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:15.954419 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:15.954426 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.955029 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.955033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:15.955137 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:15.955585 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:15.955596 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:15.957645 (+ 2049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:29:15.959922 (+ 2277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:15.959928 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:15.959933 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:15.960174 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:15.960493 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:15.960496 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:15.960498 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:15.960582 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:15.961081 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:29:15.961085 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:29:15.961195 (+ 110us) client_negotiation.cc:769] Sending connection context
0923 17:29:15.962001 (+ 806us) client_negotiation.cc:240] Negotiation successful
0923 17:29:15.962161 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260923 17:29:15.963277 2229 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.229.60" port: 42963 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:29:15.965396 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.955051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:42963 (local address 127.0.0.1:50592)
0923 17:29:15.955687 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:15.955700 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:15.955809 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:15.956044 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:15.956047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:15.956222 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:15.956294 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:15.956299 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.957062 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.957066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:15.957683 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:15.957709 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.958304 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.958308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:15.958420 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:15.958955 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:15.958975 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:15.960550 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:29:15.963595 (+ 3045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:15.963598 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:15.963600 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:15.963851 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:15.964198 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:15.964201 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:15.964203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:15.964301 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:15.964772 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:29:15.964779 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:29:15.964898 (+ 119us) client_negotiation.cc:769] Sending connection context
0923 17:29:15.965020 (+ 122us) client_negotiation.cc:240] Negotiation successful
0923 17:29:15.965168 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189}
I20260923 17:29:15.965412 2232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.955278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50592 (local address 127.11.229.60:42963)
0923 17:29:15.955387 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:15.955392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:15.955765 (+ 373us) server_negotiation.cc:407] Connection header received
0923 17:29:15.955899 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:15.955902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:15.955939 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:15.956030 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:15.956412 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.956952 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.957821 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.958190 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.958617 (+ 427us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:15.960688 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:15.960691 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:15.960694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:15.960720 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:15.963125 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:15.963982 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:15.963986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:15.963988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:15.964083 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:15.964443 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:15.964446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:15.964448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:15.964661 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:15.964768 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:15.965092 (+ 324us) server_negotiation.cc:299] Negotiation successful
0923 17:29:15.965219 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":44}
Sep 23 17:29:15 dist-test-slave-ptl9 krb5kdc[1997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184555, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.61@KRBTEST.COM for kudu/127.11.229.60@KRBTEST.COM
Sep 23 17:29:15 dist-test-slave-ptl9 krb5kdc[1997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.61@KRBTEST.COM
I20260923 17:29:15.970011 2078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09d1eec189fd4898b279d8cdd45a737c: Bootstrap starting.
Sep 23 17:29:15 dist-test-slave-ptl9 krb5kdc[1997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184555, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.60@KRBTEST.COM for kudu/127.11.229.60@KRBTEST.COM
I20260923 17:29:15.972767 2155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.959708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:42963 (local address 127.0.0.1:50594)
0923 17:29:15.959897 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:15.959912 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:15.959979 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:15.960433 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:15.960439 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:15.960610 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:15.960743 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:15.960750 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.963055 (+ 2305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.963061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:15.963721 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:15.963734 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.964381 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.964385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:15.964475 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:15.965678 (+ 1203us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:15.965692 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:15.967346 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:29:15.969953 (+ 2607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:15.969956 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:15.969958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:15.970197 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:15.970536 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:15.970539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:15.970541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:15.970628 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:15.972226 (+ 1598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:29:15.972229 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:29:15.972334 (+ 105us) client_negotiation.cc:769] Sending connection context
0923 17:29:15.972439 (+ 105us) client_negotiation.cc:240] Negotiation successful
0923 17:29:15.972581 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260923 17:29:15.973420 2078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09d1eec189fd4898b279d8cdd45a737c: Neither blocks nor log segments found. Creating new log.
I20260923 17:29:15.973462 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.959413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50594 (local address 127.11.229.60:42963)
0923 17:29:15.960180 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:15.960194 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:15.960209 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:29:15.960243 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:15.960248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:15.960291 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:15.961225 (+ 934us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:15.961362 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.962926 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.963864 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.964243 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.964377 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:15.967496 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:15.967500 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:15.967503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:15.967528 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:15.969534 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:15.970314 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:15.970320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:15.970324 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:15.970415 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:15.970756 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:15.970763 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:15.971029 (+ 266us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:15.971249 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:15.972180 (+ 931us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:15.973106 (+ 926us) server_negotiation.cc:299] Negotiation successful
0923 17:29:15.973231 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":127,"threads_started":1}
I20260923 17:29:15.973974 2157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.963817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33740 (local address 127.11.229.61:34135)
0923 17:29:15.963992 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:15.963997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:15.964148 (+ 151us) server_negotiation.cc:407] Connection header received
0923 17:29:15.964322 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:15.964328 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:15.964377 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:15.964425 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:15.965321 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.965941 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.967023 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.967294 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.967403 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:15.970513 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:15.970517 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:15.970519 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:15.970551 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:15.971986 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:15.972480 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:15.972483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:15.972484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:15.972563 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:15.972862 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:15.972867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:15.972868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:15.973078 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:15.973310 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:15.973700 (+ 390us) server_negotiation.cc:299] Negotiation successful
0923 17:29:15.973809 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"spinlock_wait_cycles":768}
I20260923 17:29:15.975016 2078 log.cc:824] T 00000000000000000000000000000000 P 09d1eec189fd4898b279d8cdd45a737c: Log is configured to *not* fsync() on all Append() calls
I20260923 17:29:15.975782 2233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.963751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:42963 (local address 127.0.0.1:50602)
0923 17:29:15.964162 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:15.964174 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:15.964243 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:15.965194 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:15.965197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:15.965371 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:15.965461 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:15.965468 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.967023 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.967026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:15.968581 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:15.968595 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.969330 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.969334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:15.969428 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:15.969887 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:15.969907 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:15.971315 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:29:15.973965 (+ 2650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:15.973967 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:15.973970 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:15.974432 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:15.974720 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:15.974723 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:15.974725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:15.974802 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:15.975225 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:29:15.975227 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:29:15.975302 (+ 75us) client_negotiation.cc:769] Sending connection context
0923 17:29:15.975475 (+ 173us) client_negotiation.cc:240] Negotiation successful
0923 17:29:15.975603 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357}
I20260923 17:29:15.976444 2235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.963977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50602 (local address 127.11.229.60:42963)
0923 17:29:15.964877 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:15.964894 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:15.964907 (+ 13us) server_negotiation.cc:407] Connection header received
0923 17:29:15.964941 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:15.964944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:15.964991 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:15.965072 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:15.965584 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.966905 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.968714 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.969089 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.969193 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:15.971447 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:15.971453 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:15.971457 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:15.971487 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:15.973847 (+ 2360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:15.974534 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:15.974540 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:15.974544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:15.974623 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:15.974894 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:15.974898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:15.974899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:15.975100 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:15.975190 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:15.976173 (+ 983us) server_negotiation.cc:299] Negotiation successful
0923 17:29:15.976284 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":93,"threads_started":1}
I20260923 17:29:15.976564 2153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25df9595488b4c52b19ac5d6ca719818: Bootstrap starting.
I20260923 17:29:15.977561 2078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09d1eec189fd4898b279d8cdd45a737c: No bootstrap required, opened a new log
I20260923 17:29:15.979259 2153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25df9595488b4c52b19ac5d6ca719818: Neither blocks nor log segments found. Creating new log.
I20260923 17:29:15.979892 2229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384fcf05ec7b40e08c8317633fecb9be: Bootstrap starting.
I20260923 17:29:15.980157 2153 log.cc:824] T 00000000000000000000000000000000 P 25df9595488b4c52b19ac5d6ca719818: Log is configured to *not* fsync() on all Append() calls
I20260923 17:29:15.980901 2078 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09d1eec189fd4898b279d8cdd45a737c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d1eec189fd4898b279d8cdd45a737c" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 33861 } } peers { permanent_uuid: "25df9595488b4c52b19ac5d6ca719818" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 34135 } } peers { permanent_uuid: "384fcf05ec7b40e08c8317633fecb9be" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 42963 } }
I20260923 17:29:15.981176 2078 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09d1eec189fd4898b279d8cdd45a737c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09d1eec189fd4898b279d8cdd45a737c, State: Initialized, Role: FOLLOWER
I20260923 17:29:15.981520 2078 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09d1eec189fd4898b279d8cdd45a737c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d1eec189fd4898b279d8cdd45a737c" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 33861 } } peers { permanent_uuid: "25df9595488b4c52b19ac5d6ca719818" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 34135 } } peers { permanent_uuid: "384fcf05ec7b40e08c8317633fecb9be" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 42963 } }
I20260923 17:29:15.981993 2238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d1eec189fd4898b279d8cdd45a737c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d1eec189fd4898b279d8cdd45a737c" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 33861 } } peers { permanent_uuid: "25df9595488b4c52b19ac5d6ca719818" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 34135 } } peers { permanent_uuid: "384fcf05ec7b40e08c8317633fecb9be" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 42963 } } }
I20260923 17:29:15.982182 2238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d1eec189fd4898b279d8cdd45a737c [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:29:15.982515 2153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25df9595488b4c52b19ac5d6ca719818: No bootstrap required, opened a new log
I20260923 17:29:15.982553 2229 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 384fcf05ec7b40e08c8317633fecb9be: Neither blocks nor log segments found. Creating new log.
I20260923 17:29:15.982851 2078 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09d1eec189fd4898b279d8cdd45a737c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:29:15.983328 2229 log.cc:824] T 00000000000000000000000000000000 P 384fcf05ec7b40e08c8317633fecb9be: Log is configured to *not* fsync() on all Append() calls
I20260923 17:29:15.984951 12180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:29:15.986326 2052 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "09d1eec189fd4898b279d8cdd45a737c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51580
I20260923 17:29:15.986553 2052 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09d1eec189fd4898b279d8cdd45a737c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 17:29:15.986660 2052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09d1eec189fd4898b279d8cdd45a737c [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:29:15.986686 2153 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25df9595488b4c52b19ac5d6ca719818 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d1eec189fd4898b279d8cdd45a737c" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 33861 } } peers { permanent_uuid: "25df9595488b4c52b19ac5d6ca719818" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 34135 } } peers { permanent_uuid: "384fcf05ec7b40e08c8317633fecb9be" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 42963 } }
I20260923 17:29:15.986944 2153 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25df9595488b4c52b19ac5d6ca719818 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25df9595488b4c52b19ac5d6ca719818, State: Initialized, Role: FOLLOWER
I20260923 17:29:15.987282 2153 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25df9595488b4c52b19ac5d6ca719818 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d1eec189fd4898b279d8cdd45a737c" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 33861 } } peers { permanent_uuid: "25df9595488b4c52b19ac5d6ca719818" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 34135 } } peers { permanent_uuid: "384fcf05ec7b40e08c8317633fecb9be" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 42963 } }
I20260923 17:29:15.988027 2052 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09d1eec189fd4898b279d8cdd45a737c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d1eec189fd4898b279d8cdd45a737c" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 33861 } } peers { permanent_uuid: "25df9595488b4c52b19ac5d6ca719818" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 34135 } } peers { permanent_uuid: "384fcf05ec7b40e08c8317633fecb9be" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 42963 } }
I20260923 17:29:15.988629 2153 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25df9595488b4c52b19ac5d6ca719818 [sys.catalog]: configured and running, proceeding with master startup.
W20260923 17:29:15.988873 2250 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 09d1eec189fd4898b279d8cdd45a737c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:29:15.988983 2250 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:29:15.988965 2052 leader_election.cc:288] T 00000000000000000000000000000000 P 09d1eec189fd4898b279d8cdd45a737c [CANDIDATE]: Term 1 election: Requested vote from peers 25df9595488b4c52b19ac5d6ca719818 (127.11.229.61:34135), 384fcf05ec7b40e08c8317633fecb9be (127.11.229.60:42963)
I20260923 17:29:15.987773 2010 master_runner.cc:186] Error getting master registration for 127.11.229.61:34135: OK, instance_id { permanent_uuid: "25df9595488b4c52b19ac5d6ca719818" instance_seqno: 1790184555704029 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 17:29:15.991272 2010 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 17:29:15.993011 2229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384fcf05ec7b40e08c8317633fecb9be: No bootstrap required, opened a new log
I20260923 17:29:15.991729 2252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25df9595488b4c52b19ac5d6ca719818 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d1eec189fd4898b279d8cdd45a737c" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 33861 } } peers { permanent_uuid: "25df9595488b4c52b19ac5d6ca719818" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 34135 } } peers { permanent_uuid: "384fcf05ec7b40e08c8317633fecb9be" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 42963 } } }
I20260923 17:29:15.999560 2252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25df9595488b4c52b19ac5d6ca719818 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:29:15.992238 2126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "09d1eec189fd4898b279d8cdd45a737c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "25df9595488b4c52b19ac5d6ca719818"
I20260923 17:29:15.999943 2126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25df9595488b4c52b19ac5d6ca719818 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:29:15.999965 2229 raft_consensus.cc:348] T 00000000000000000000000000000000 P 384fcf05ec7b40e08c8317633fecb9be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d1eec189fd4898b279d8cdd45a737c" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 33861 } } peers { permanent_uuid: "25df9595488b4c52b19ac5d6ca719818" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 34135 } } peers { permanent_uuid: "384fcf05ec7b40e08c8317633fecb9be" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 42963 } }
I20260923 17:29:16.000559 2229 raft_consensus.cc:729] T 00000000000000000000000000000000 P 384fcf05ec7b40e08c8317633fecb9be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 384fcf05ec7b40e08c8317633fecb9be, State: Initialized, Role: FOLLOWER
I20260923 17:29:16.001173 2229 consensus_queue.cc:260] T 00000000000000000000000000000000 P 384fcf05ec7b40e08c8317633fecb9be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d1eec189fd4898b279d8cdd45a737c" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 33861 } } peers { permanent_uuid: "25df9595488b4c52b19ac5d6ca719818" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 34135 } } peers { permanent_uuid: "384fcf05ec7b40e08c8317633fecb9be" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 42963 } }
I20260923 17:29:16.001401 2126 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 25df9595488b4c52b19ac5d6ca719818 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09d1eec189fd4898b279d8cdd45a737c in term 1.
I20260923 17:29:16.001878 2029 leader_election.cc:302] T 00000000000000000000000000000000 P 09d1eec189fd4898b279d8cdd45a737c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09d1eec189fd4898b279d8cdd45a737c, 25df9595488b4c52b19ac5d6ca719818; no voters:
I20260923 17:29:16.002226 2238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09d1eec189fd4898b279d8cdd45a737c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:29:16.002492 2238 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09d1eec189fd4898b279d8cdd45a737c [term 1 LEADER]: Becoming Leader. State: Replica: 09d1eec189fd4898b279d8cdd45a737c, State: Running, Role: LEADER
I20260923 17:29:16.002873 2238 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09d1eec189fd4898b279d8cdd45a737c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d1eec189fd4898b279d8cdd45a737c" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 33861 } } peers { permanent_uuid: "25df9595488b4c52b19ac5d6ca719818" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 34135 } } peers { permanent_uuid: "384fcf05ec7b40e08c8317633fecb9be" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 42963 } }
I20260923 17:29:16.004140 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "09d1eec189fd4898b279d8cdd45a737c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "384fcf05ec7b40e08c8317633fecb9be"
I20260923 17:29:16.004400 2203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 384fcf05ec7b40e08c8317633fecb9be [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:29:16.004931 2238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d1eec189fd4898b279d8cdd45a737c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09d1eec189fd4898b279d8cdd45a737c. Latest consensus state: current_term: 1 leader_uuid: "09d1eec189fd4898b279d8cdd45a737c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d1eec189fd4898b279d8cdd45a737c" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 33861 } } peers { permanent_uuid: "25df9595488b4c52b19ac5d6ca719818" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 34135 } } peers { permanent_uuid: "384fcf05ec7b40e08c8317633fecb9be" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 42963 } } }
I20260923 17:29:16.005069 2238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d1eec189fd4898b279d8cdd45a737c [sys.catalog]: This master's current role is: LEADER
I20260923 17:29:16.005826 2203 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 384fcf05ec7b40e08c8317633fecb9be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09d1eec189fd4898b279d8cdd45a737c in term 1.
I20260923 17:29:16.006160 2229 sys_catalog.cc:564] T 00000000000000000000000000000000 P 384fcf05ec7b40e08c8317633fecb9be [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:29:16.013441 2272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 384fcf05ec7b40e08c8317633fecb9be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d1eec189fd4898b279d8cdd45a737c" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 33861 } } peers { permanent_uuid: "25df9595488b4c52b19ac5d6ca719818" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 34135 } } peers { permanent_uuid: "384fcf05ec7b40e08c8317633fecb9be" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 42963 } } }
I20260923 17:29:16.013604 2272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 384fcf05ec7b40e08c8317633fecb9be [sys.catalog]: This master's current role is: FOLLOWER
W20260923 17:29:15.994199 2267 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 25df9595488b4c52b19ac5d6ca719818: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:29:16.017539 2267 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:29:16.019289 2275 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:29:16.020177 2275 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:29:16.022876 2157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.991743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33742 (local address 127.11.229.61:34135)
0923 17:29:15.997828 (+ 6085us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:15.997833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:15.997852 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:29:15.997901 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:15.997904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:15.997957 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:15.998046 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:16.006534 (+ 8488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:16.007325 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:16.010833 (+ 3508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:16.011186 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:16.011278 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:16.019191 (+ 7913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:16.019195 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:16.019198 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:16.019225 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:16.021005 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:16.021524 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:16.021528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:16.021529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:16.021610 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:16.021881 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:16.021884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:16.021886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:16.022090 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:16.022215 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:16.022584 (+ 369us) server_negotiation.cc:299] Negotiation successful
0923 17:29:16.022715 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5970}
I20260923 17:29:16.023725 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.998270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50608 (local address 127.11.229.60:42963)
0923 17:29:15.998396 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:15.998400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:16.008963 (+ 10563us) server_negotiation.cc:407] Connection header received
0923 17:29:16.009140 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:16.009143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:16.009197 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:16.009249 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:16.010315 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:16.011022 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:16.012184 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:16.012583 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:16.012658 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:16.014761 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:16.014765 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:16.014767 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:16.014793 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:16.016609 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:16.017093 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:16.017097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:16.017099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:16.017181 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:16.017827 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:16.017830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:16.017832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:16.018052 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:16.018187 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:16.023442 (+ 5255us) server_negotiation.cc:299] Negotiation successful
0923 17:29:16.023568 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260923 17:29:16.029302 2286 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 384fcf05ec7b40e08c8317633fecb9be: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:29:16.029458 2286 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:29:16.030107 2081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.997763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51592 (local address 127.11.229.62:33861)
0923 17:29:15.997915 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:15.997922 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:16.008618 (+ 10696us) server_negotiation.cc:407] Connection header received
0923 17:29:16.008772 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:16.008776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:16.008832 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:16.008932 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:16.013761 (+ 4829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:16.014600 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:16.024338 (+ 9738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:16.024707 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:16.024783 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:16.026537 (+ 1754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:16.026541 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:16.026544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:16.026571 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:16.028261 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:16.028749 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:16.028752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:16.028754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:16.028833 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:16.029108 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:16.029111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:16.029113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:16.029365 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:16.029472 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:16.029812 (+ 340us) server_negotiation.cc:299] Negotiation successful
0923 17:29:16.029941 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 17:29:16.034509 2203 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 384fcf05ec7b40e08c8317633fecb9be [term 1 FOLLOWER]: Refusing update from remote peer 09d1eec189fd4898b279d8cdd45a737c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:29:16.034518 2126 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 25df9595488b4c52b19ac5d6ca719818 [term 1 FOLLOWER]: Refusing update from remote peer 09d1eec189fd4898b279d8cdd45a737c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:29:16.035173 2238 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 09d1eec189fd4898b279d8cdd45a737c [LEADER]: Connected to new peer: Peer: permanent_uuid: "384fcf05ec7b40e08c8317633fecb9be" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 42963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:29:16.035619 2238 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 09d1eec189fd4898b279d8cdd45a737c [LEADER]: Connected to new peer: Peer: permanent_uuid: "25df9595488b4c52b19ac5d6ca719818" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 34135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:29:16.039613 2272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 384fcf05ec7b40e08c8317633fecb9be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09d1eec189fd4898b279d8cdd45a737c. Latest consensus state: current_term: 1 leader_uuid: "09d1eec189fd4898b279d8cdd45a737c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d1eec189fd4898b279d8cdd45a737c" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 33861 } } peers { permanent_uuid: "25df9595488b4c52b19ac5d6ca719818" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 34135 } } peers { permanent_uuid: "384fcf05ec7b40e08c8317633fecb9be" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 42963 } } }
I20260923 17:29:16.039635 2252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25df9595488b4c52b19ac5d6ca719818 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09d1eec189fd4898b279d8cdd45a737c. Latest consensus state: current_term: 1 leader_uuid: "09d1eec189fd4898b279d8cdd45a737c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d1eec189fd4898b279d8cdd45a737c" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 33861 } } peers { permanent_uuid: "25df9595488b4c52b19ac5d6ca719818" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 34135 } } peers { permanent_uuid: "384fcf05ec7b40e08c8317633fecb9be" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 42963 } } }
I20260923 17:29:16.039742 2272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 384fcf05ec7b40e08c8317633fecb9be [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:29:16.039776 2252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25df9595488b4c52b19ac5d6ca719818 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:29:16.046023 2273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d1eec189fd4898b279d8cdd45a737c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "09d1eec189fd4898b279d8cdd45a737c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d1eec189fd4898b279d8cdd45a737c" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 33861 } } peers { permanent_uuid: "25df9595488b4c52b19ac5d6ca719818" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 34135 } } peers { permanent_uuid: "384fcf05ec7b40e08c8317633fecb9be" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 42963 } } }
I20260923 17:29:16.046169 2273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d1eec189fd4898b279d8cdd45a737c [sys.catalog]: This master's current role is: LEADER
I20260923 17:29:16.046453 2288 mvcc.cc:205] Tried to move back new op lower bound from 7332595941486161920 to 7332595941391667200. Current Snapshot: MvccSnapshot[applied={T|T < 7332595941486161920}]
I20260923 17:29:16.046417 2272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 384fcf05ec7b40e08c8317633fecb9be [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "09d1eec189fd4898b279d8cdd45a737c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d1eec189fd4898b279d8cdd45a737c" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 33861 } } peers { permanent_uuid: "25df9595488b4c52b19ac5d6ca719818" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 34135 } } peers { permanent_uuid: "384fcf05ec7b40e08c8317633fecb9be" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 42963 } } }
I20260923 17:29:16.046522 2272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 384fcf05ec7b40e08c8317633fecb9be [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:29:16.052230 2238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d1eec189fd4898b279d8cdd45a737c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "09d1eec189fd4898b279d8cdd45a737c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d1eec189fd4898b279d8cdd45a737c" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 33861 } } peers { permanent_uuid: "25df9595488b4c52b19ac5d6ca719818" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 34135 } } peers { permanent_uuid: "384fcf05ec7b40e08c8317633fecb9be" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 42963 } } }
I20260923 17:29:16.052361 2238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d1eec189fd4898b279d8cdd45a737c [sys.catalog]: This master's current role is: LEADER
I20260923 17:29:16.053570 2252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25df9595488b4c52b19ac5d6ca719818 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "09d1eec189fd4898b279d8cdd45a737c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d1eec189fd4898b279d8cdd45a737c" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 33861 } } peers { permanent_uuid: "25df9595488b4c52b19ac5d6ca719818" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 34135 } } peers { permanent_uuid: "384fcf05ec7b40e08c8317633fecb9be" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 42963 } } }
I20260923 17:29:16.053682 2252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25df9595488b4c52b19ac5d6ca719818 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:29:16.053937 2290 mvcc.cc:205] Tried to move back new op lower bound from 7332595941486161920 to 7332595941391667200. Current Snapshot: MvccSnapshot[applied={T|T < 7332595941486161920}]
I20260923 17:29:16.056224 2275 catalog_manager.cc:1347] Generated new cluster ID: f4f719b6b0d64b4fb2d3e7004616a533
I20260923 17:29:16.056285 2275 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:29:16.103499 2275 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:29:16.104837 2275 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:29:16.122193 2275 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09d1eec189fd4898b279d8cdd45a737c: Generated new TSK 0
I20260923 17:29:16.123203 2275 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 17:29:16.161540 2157 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:16.158069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33746 (local address 127.11.229.61:34135)
0923 17:29:16.158214 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:16.158220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:16.160303 (+ 2083us) server_negotiation.cc:407] Connection header received
0923 17:29:16.160559 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:16.160564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:16.160625 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:16.160686 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:16.161390 (+ 704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33746: BlockingRecv error: recv got EOF from 127.0.0.1:33746 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260923 17:29:16.163030 2081 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:16.158991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51604 (local address 127.11.229.62:33861)
0923 17:29:16.159113 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:16.159118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:16.162065 (+ 2947us) server_negotiation.cc:407] Connection header received
0923 17:29:16.162249 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:16.162252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:16.162303 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:16.162355 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:16.162901 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51604: BlockingRecv error: recv got EOF from 127.0.0.1:51604 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260923 17:29:16.164637 2234 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:16.159683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50620 (local address 127.11.229.60:42963)
0923 17:29:16.159804 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:16.159809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:16.163522 (+ 3713us) server_negotiation.cc:407] Connection header received
0923 17:29:16.163834 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:16.163838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:16.163896 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:16.163954 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:16.164514 (+ 560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50620: BlockingRecv error: recv got EOF from 127.0.0.1:50620 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260923 17:29:16.169723 2157 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:16.167915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33750 (local address 127.11.229.61:34135)
0923 17:29:16.168036 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:16.168040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:16.168618 (+ 578us) server_negotiation.cc:407] Connection header received
0923 17:29:16.168861 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:16.168864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:16.168951 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:16.169011 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:16.169611 (+ 600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33750: BlockingRecv error: recv got EOF from 127.0.0.1:33750 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260923 17:29:16.174888 2157 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:16.173198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33764 (local address 127.11.229.61:34135)
0923 17:29:16.173355 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:16.173359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:16.173909 (+ 550us) server_negotiation.cc:407] Connection header received
0923 17:29:16.174117 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:16.174120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:16.174175 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:16.174232 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:16.174781 (+ 549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33764: BlockingRecv error: recv got EOF from 127.0.0.1:33764 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260923 17:29:16.180438 2157 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:16.178872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33774 (local address 127.11.229.61:34135)
0923 17:29:16.178979 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:16.178983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:16.179543 (+ 560us) server_negotiation.cc:407] Connection header received
0923 17:29:16.179730 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:16.179733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:16.179781 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:16.179830 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:16.180320 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33774: BlockingRecv error: recv got EOF from 127.0.0.1:33774 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260923 17:29:16.188050 2157 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:16.186500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33776 (local address 127.11.229.61:34135)
0923 17:29:16.186629 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:16.186633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:16.187148 (+ 515us) server_negotiation.cc:407] Connection header received
0923 17:29:16.187325 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:16.187328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:16.187393 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:16.187449 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:16.187947 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33776: BlockingRecv error: recv got EOF from 127.0.0.1:33776 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260923 17:29:16.199883 2157 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:16.198276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33780 (local address 127.11.229.61:34135)
0923 17:29:16.198404 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:16.198408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:16.198949 (+ 541us) server_negotiation.cc:407] Connection header received
0923 17:29:16.199146 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:16.199150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:16.199199 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:16.199249 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:16.199761 (+ 512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33780: BlockingRecv error: recv got EOF from 127.0.0.1:33780 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260923 17:29:16.219367 2157 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:16.218219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33796 (local address 127.11.229.61:34135)
0923 17:29:16.218365 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:16.218370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:16.218515 (+ 145us) server_negotiation.cc:407] Connection header received
0923 17:29:16.218649 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:16.218653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:16.218716 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:16.218813 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:16.219243 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33796: BlockingRecv error: recv got EOF from 127.0.0.1:33796 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260923 17:29:16.254575 2157 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:16.253465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33802 (local address 127.11.229.61:34135)
0923 17:29:16.253630 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:16.253634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:16.253708 (+ 74us) server_negotiation.cc:407] Connection header received
0923 17:29:16.253833 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:16.253835 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:16.253886 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:16.253958 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:16.254416 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33802: BlockingRecv error: recv got EOF from 127.0.0.1:33802 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260923 17:29:16.321728 2157 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:16.320560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33816 (local address 127.11.229.61:34135)
0923 17:29:16.320715 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:16.320719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:16.320854 (+ 135us) server_negotiation.cc:407] Connection header received
0923 17:29:16.320979 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:16.320981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:16.321037 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:16.321124 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:16.321630 (+ 506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33816: BlockingRecv error: recv got EOF from 127.0.0.1:33816 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260923 17:29:16.453207 2157 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:16.451924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33822 (local address 127.11.229.61:34135)
0923 17:29:16.452105 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:16.452111 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:16.452257 (+ 146us) server_negotiation.cc:407] Connection header received
0923 17:29:16.452448 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:16.452451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:16.452508 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:16.452606 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:16.453058 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33822: BlockingRecv error: recv got EOF from 127.0.0.1:33822 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260923 17:29:16.712527 2157 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:16.711486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33824 (local address 127.11.229.61:34135)
0923 17:29:16.711625 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:16.711630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:16.711714 (+ 84us) server_negotiation.cc:407] Connection header received
0923 17:29:16.711836 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:16.711839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:16.711895 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:16.711999 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:16.712418 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33824: BlockingRecv error: recv got EOF from 127.0.0.1:33824 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 17:29:17.018965 2267 catalog_manager.cc:1259] Loaded cluster ID: f4f719b6b0d64b4fb2d3e7004616a533
I20260923 17:29:17.019050 2267 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 25df9595488b4c52b19ac5d6ca719818: loading cluster ID for follower catalog manager: success
I20260923 17:29:17.020731 2267 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 25df9595488b4c52b19ac5d6ca719818: acquiring CA information for follower catalog manager: success
I20260923 17:29:17.021900 2267 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 25df9595488b4c52b19ac5d6ca719818: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 17:29:17.030728 2286 catalog_manager.cc:1259] Loaded cluster ID: f4f719b6b0d64b4fb2d3e7004616a533
I20260923 17:29:17.030838 2286 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 384fcf05ec7b40e08c8317633fecb9be: loading cluster ID for follower catalog manager: success
I20260923 17:29:17.032604 2286 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 384fcf05ec7b40e08c8317633fecb9be: acquiring CA information for follower catalog manager: success
I20260923 17:29:17.033902 2286 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 384fcf05ec7b40e08c8317633fecb9be: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 17:29:17.230751 2362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:17.226567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33838 (local address 127.11.229.61:34135)
0923 17:29:17.226849 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:17.226856 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:17.226881 (+ 25us) server_negotiation.cc:407] Connection header received
0923 17:29:17.227086 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:17.227097 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:17.227229 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:17.227354 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:29:17.227667 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:17.228246 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:17.229122 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:17.229424 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:17.229542 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:17.229888 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:29:17.230332 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:29:17.230469 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:17.230525 (+ 56us) server_negotiation.cc:299] Negotiation successful
0923 17:29:17.230583 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":67,"threads_started":1}
I20260923 17:29:17.237978 2368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:17.234092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50628 (local address 127.11.229.60:42963)
0923 17:29:17.234351 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:17.234356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:17.234371 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:29:17.234550 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:17.234552 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:17.234671 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:17.234799 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:29:17.235027 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:17.235555 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:17.236406 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:17.236643 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:17.236757 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:17.237072 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:29:17.237574 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:29:17.237712 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:17.237755 (+ 43us) server_negotiation.cc:299] Negotiation successful
0923 17:29:17.237815 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":64,"threads_started":1}
I20260923 17:29:17.245153 2362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:17.240965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33840 (local address 127.11.229.61:34135)
0923 17:29:17.241123 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:17.241127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:17.241410 (+ 283us) server_negotiation.cc:407] Connection header received
0923 17:29:17.241604 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:17.241607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:17.241658 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:17.241706 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:29:17.242265 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:17.242836 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:17.243943 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:17.244266 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:17.244362 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:17.244616 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:29:17.244717 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:29:17.244786 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:17.244928 (+ 142us) server_negotiation.cc:299] Negotiation successful
0923 17:29:17.244985 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 17:29:17.245496 2375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:17.240895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51618 (local address 127.11.229.62:33861)
0923 17:29:17.241161 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:17.241166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:17.241413 (+ 247us) server_negotiation.cc:407] Connection header received
0923 17:29:17.241516 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:17.241522 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:17.241689 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:17.241815 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:29:17.242091 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:17.242685 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:17.243594 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:17.243960 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:17.244071 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:17.244385 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:29:17.244743 (+ 358us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:29:17.244862 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:17.245169 (+ 307us) server_negotiation.cc:299] Negotiation successful
0923 17:29:17.245250 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":65,"threads_started":1}
I20260923 17:29:17.246016 2368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:17.241523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50638 (local address 127.11.229.60:42963)
0923 17:29:17.241629 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:17.241633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:17.241647 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:29:17.241693 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:17.241696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:17.241745 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:17.241834 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:29:17.242810 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:17.243576 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:17.244667 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:17.244992 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:17.245088 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:17.245476 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:29:17.245591 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:29:17.245650 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:17.245797 (+ 147us) server_negotiation.cc:299] Negotiation successful
0923 17:29:17.245849 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 17:29:17.248539 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 2010
I20260923 17:29:17.255622 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 2085
I20260923 17:29:17.261992 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 2161
2026-09-23T17:29:17Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3128 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:29:17 dist-test-slave-ptl9 krb5kdc[2379](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:29:17 dist-test-slave-ptl9 krb5kdc[2379](info): set up 2 sockets
Sep 23 17:29:17 dist-test-slave-ptl9 krb5kdc[2379](info): commencing operation
krb5kdc: starting...
W20260923 17:29:18.309916 12180 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/krb5kdc/test-user.keytab.
Sep 23 17:29:18 dist-test-slave-ptl9 krb5kdc[2379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:29:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:29:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.229.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
Entry for principal kudu/127.11.229.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab.
I20260923 17:29:18.433647 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:34985
--webserver_interface=127.11.229.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:44703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:34985,127.11.229.61:41487,127.11.229.60:35841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:29:18.548250 2392 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 17:29:18.548532 2392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:18.548555 2392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:18.548591 2392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:29:18.552604 2392 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:29:18.552657 2392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:18.552676 2392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:18.552692 2392 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:29:18.552736 2392 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:29:18.558375 2392 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:44703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:34985,127.11.229.61:41487,127.11.229.60:35841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.62.keytab
--principal=kudu/127.11.229.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.62:34985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.229.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:29:18.559460 2392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:18.560514 2392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:18.568832 2397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:18.568910 2398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:18.568959 2400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:18.569097 2392 server_base.cc:1034] running on GCE node
I20260923 17:29:18.569772 2392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:18.570868 2392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:18.572028 2392 hybrid_clock.cc:648] HybridClock initialized: now 1790184558572002 us; error 37 us; skew 500 ppm
Sep 23 17:29:18 dist-test-slave-ptl9 krb5kdc[2379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184558, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:29:18.575022 2392 init.cc:377] Logged in from keytab as kudu/127.11.229.62@KRBTEST.COM (short username kudu)
I20260923 17:29:18.575908 2392 webserver.cc:460] Webserver started at http://127.11.229.62:37749/ using document root <none> and password file <none>
I20260923 17:29:18.576474 2392 fs_manager.cc:359] Metadata directory not provided
I20260923 17:29:18.576568 2392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:29:18.576803 2392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:29:18.578882 2392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-0/data/instance:
uuid: "3655778989524389bbb940717abeb9e3"
format_stamp: "Formatted at 2026-09-23 17:29:18 on dist-test-slave-ptl9"
server_key: "c2dd62aeeb173501e8f348f91645155b"
server_key_iv: "1bba77fc1beef10be65d15e67da3e194"
server_key_version: "encryptionkey@0"
I20260923 17:29:18.579459 2392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-0/wal/instance:
uuid: "3655778989524389bbb940717abeb9e3"
format_stamp: "Formatted at 2026-09-23 17:29:18 on dist-test-slave-ptl9"
server_key: "c2dd62aeeb173501e8f348f91645155b"
server_key_iv: "1bba77fc1beef10be65d15e67da3e194"
server_key_version: "encryptionkey@0"
I20260923 17:29:18.583112 2392 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260923 17:29:18.585479 2407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:18.586653 2392 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:29:18.586797 2392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-0/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-0/wal
uuid: "3655778989524389bbb940717abeb9e3"
format_stamp: "Formatted at 2026-09-23 17:29:18 on dist-test-slave-ptl9"
server_key: "c2dd62aeeb173501e8f348f91645155b"
server_key_iv: "1bba77fc1beef10be65d15e67da3e194"
server_key_version: "encryptionkey@0"
I20260923 17:29:18.586913 2392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:29:18.598440 2392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:29:18.601490 2392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:29:18.601694 2392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:29:18.609366 2392 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.62:34985
I20260923 17:29:18.609659 2459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.62:34985 every 8 connection(s)
I20260923 17:29:18.610466 2392 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-0/data/info.pb
I20260923 17:29:18.613368 2460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:29:18.617117 2460 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34985 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:29:18.619858 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 2392
I20260923 17:29:18.620007 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-0/wal/instance
I20260923 17:29:18.620330 12180 external_mini_cluster.cc:1428] Setting key e8f74884c13d1f2bc2d962d33c6f3f71
Sep 23 17:29:18 dist-test-slave-ptl9 krb5kdc[2379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184558, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.229.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.61@KRBTEST.COM" created.
I20260923 17:29:18.633594 2462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:18.619529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34985 (local address 127.0.0.1:50832)
0923 17:29:18.620143 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:18.620209 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:18.621497 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:18.622309 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:18.622314 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:18.622648 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:18.622892 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:18.622905 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:18.624662 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:18.624665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:18.625565 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:18.625575 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:18.626527 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:18.626530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:18.626622 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:18.627263 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:18.627279 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:18.628713 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:29:18.630696 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:18.630704 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:18.630712 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:18.630975 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:18.631309 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:18.631311 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:18.631313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:18.631424 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:18.632122 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:29:18.632132 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:29:18.632283 (+ 151us) client_negotiation.cc:769] Sending connection context
0923 17:29:18.632433 (+ 150us) client_negotiation.cc:240] Negotiation successful
0923 17:29:18.632704 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":2,"thread_start_us":170,"threads_started":1}
I20260923 17:29:18.633599 2463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:18.619529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50832 (local address 127.11.229.62:34985)
0923 17:29:18.620249 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:18.620260 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:18.620305 (+ 45us) server_negotiation.cc:407] Connection header received
0923 17:29:18.621830 (+ 1525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:18.621835 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:18.622084 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:18.622295 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:18.623059 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:18.624537 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:18.625691 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:18.626371 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:18.626590 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:18.628881 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:18.628891 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:18.628909 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:18.628945 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:18.630590 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:18.631099 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:18.631112 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:18.631114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:18.631222 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:18.631515 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:18.631517 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:18.631518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:18.632002 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:18.632110 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:18.632547 (+ 437us) server_negotiation.cc:299] Negotiation successful
0923 17:29:18.632812 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":129,"server-negotiator.queue_time_us":599,"spinlock_wait_cycles":77696,"thread_start_us":170,"threads_started":1}
I20260923 17:29:18.636435 2460 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41487 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:29:18.637243 2462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:18.636982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:41487 (local address 127.0.0.1:59282)
0923 17:29:18.637117 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:18.637184 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.61:41487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260923 17:29:18.637481 2410 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.229.61:41487: connect: Connection refused (error 111)
I20260923 17:29:18.638441 2462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:18.638177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:41487 (local address 127.0.0.1:59290)
0923 17:29:18.638302 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:18.638360 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.61:41487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260923 17:29:18.638594 2460 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.229.61:41487: Network error: Client connection negotiation failed: client connection to 127.11.229.61:41487: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.61.keytab.
Entry for principal kudu/127.11.229.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.61.keytab.
I20260923 17:29:18.649003 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.61:41487
--webserver_interface=127.11.229.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.61.keytab
--principal=kudu/127.11.229.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:44703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:34985,127.11.229.61:41487,127.11.229.60:35841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:29:18.687884 2460 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41487 } attempt: 1
I20260923 17:29:18.688864 2462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:18.688510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:41487 (local address 127.0.0.1:59306)
0923 17:29:18.688693 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:18.688768 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.61:41487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260923 17:29:18.689739 2462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:18.689490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:41487 (local address 127.0.0.1:59318)
0923 17:29:18.689616 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:18.689670 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.61:41487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 17:29:18.689939 2460 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.229.61:41487: Network error: Client connection negotiation failed: client connection to 127.11.229.61:41487: connect: Connection refused (error 111)
I20260923 17:29:18.758167 2460 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41487 } attempt: 2
I20260923 17:29:18.759088 2462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:18.758710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:41487 (local address 127.0.0.1:59330)
0923 17:29:18.758901 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:18.758991 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.61:41487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260923 17:29:18.759855 2462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:18.759606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:41487 (local address 127.0.0.1:59332)
0923 17:29:18.759721 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:18.759777 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.61:41487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260923 17:29:18.760036 2460 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.229.61:41487: Network error: Client connection negotiation failed: client connection to 127.11.229.61:41487: connect: Connection refused (error 111)
W20260923 17:29:18.777388 2467 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 17:29:18.777715 2467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:18.777781 2467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:18.777860 2467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:29:18.781901 2467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:29:18.781976 2467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:18.782021 2467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:18.782079 2467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:29:18.782117 2467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:29:18.787397 2467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:44703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:34985,127.11.229.61:41487,127.11.229.60:35841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.61.keytab
--principal=kudu/127.11.229.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.61:41487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.229.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:29:18.788471 2467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:18.789525 2467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:18.798038 2473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:18.798040 2472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:18.798022 2475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:18.798429 2467 server_base.cc:1034] running on GCE node
I20260923 17:29:18.798921 2467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:18.800025 2467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:18.801201 2467 hybrid_clock.cc:648] HybridClock initialized: now 1790184558801186 us; error 24 us; skew 500 ppm
Sep 23 17:29:18 dist-test-slave-ptl9 krb5kdc[2379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184558, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:29:18.804008 2467 init.cc:377] Logged in from keytab as kudu/127.11.229.61@KRBTEST.COM (short username kudu)
I20260923 17:29:18.804929 2467 webserver.cc:460] Webserver started at http://127.11.229.61:40947/ using document root <none> and password file <none>
I20260923 17:29:18.805548 2467 fs_manager.cc:359] Metadata directory not provided
I20260923 17:29:18.805611 2467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:29:18.805819 2467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:29:18.807857 2467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-1/data/instance:
uuid: "e254b9e5625e4bb7b8202b366208cb94"
format_stamp: "Formatted at 2026-09-23 17:29:18 on dist-test-slave-ptl9"
server_key: "481200f48da0f79275bacecfd3fe3a6f"
server_key_iv: "2b3c1b08dee268862815e85ea215af09"
server_key_version: "encryptionkey@0"
I20260923 17:29:18.808418 2467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-1/wal/instance:
uuid: "e254b9e5625e4bb7b8202b366208cb94"
format_stamp: "Formatted at 2026-09-23 17:29:18 on dist-test-slave-ptl9"
server_key: "481200f48da0f79275bacecfd3fe3a6f"
server_key_iv: "2b3c1b08dee268862815e85ea215af09"
server_key_version: "encryptionkey@0"
I20260923 17:29:18.812172 2467 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260923 17:29:18.814704 2482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:18.815915 2467 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:29:18.816053 2467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-1/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-1/wal
uuid: "e254b9e5625e4bb7b8202b366208cb94"
format_stamp: "Formatted at 2026-09-23 17:29:18 on dist-test-slave-ptl9"
server_key: "481200f48da0f79275bacecfd3fe3a6f"
server_key_iv: "2b3c1b08dee268862815e85ea215af09"
server_key_version: "encryptionkey@0"
I20260923 17:29:18.816165 2467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:29:18.851222 2460 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41487 } attempt: 3
I20260923 17:29:18.852154 2462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:18.851804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:41487 (local address 127.0.0.1:59334)
0923 17:29:18.851986 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:18.852058 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.61:41487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260923 17:29:18.852949 2462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:18.852736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:41487 (local address 127.0.0.1:59350)
0923 17:29:18.852844 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:18.852895 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.61:41487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260923 17:29:18.853104 2460 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.229.61:41487: Network error: Client connection negotiation failed: client connection to 127.11.229.61:41487: connect: Connection refused (error 111)
I20260923 17:29:18.858412 2467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:29:18.861783 2467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:29:18.861994 2467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:29:18.869045 2467 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.61:41487
I20260923 17:29:18.869349 2534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.61:41487 every 8 connection(s)
I20260923 17:29:18.870133 2467 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-1/data/info.pb
I20260923 17:29:18.872810 2535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:29:18.875772 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 2467
I20260923 17:29:18.875891 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-1/wal/instance
I20260923 17:29:18.876223 12180 external_mini_cluster.cc:1428] Setting key 62382adea78addb85f90e4e5f9d41045
I20260923 17:29:18.876775 2535 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34985 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.11.229.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.229.60@KRBTEST.COM" created.
Sep 23 17:29:18 dist-test-slave-ptl9 krb5kdc[2379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184558, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.61@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
I20260923 17:29:18.892071 2463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:18.878746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50838 (local address 127.11.229.62:34985)
0923 17:29:18.878882 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:18.878886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:18.879874 (+ 988us) server_negotiation.cc:407] Connection header received
0923 17:29:18.880900 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:18.880907 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:18.880985 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:18.881103 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:18.882300 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:18.883206 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:18.884434 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:18.884676 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:18.884761 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:18.887961 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:18.887968 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:18.887974 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:18.888010 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:18.889589 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:18.890246 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:18.890248 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:18.890249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:18.890326 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:18.890741 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:18.890743 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:18.890744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:18.890952 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:18.891115 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:18.891744 (+ 629us) server_negotiation.cc:299] Negotiation successful
0923 17:29:18.891888 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 17:29:18.892848 2538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:18.879202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34985 (local address 127.0.0.1:50838)
0923 17:29:18.879681 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:18.879734 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:18.880675 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:18.881254 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:18.881309 (+ 55us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:18.881685 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:18.882146 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:18.882159 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:18.883337 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:18.883340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:18.884297 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:18.884307 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:18.884811 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:18.884813 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:18.884948 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:18.885550 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:18.885565 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:18.887789 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:29:18.889707 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:18.889716 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:18.889736 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:18.890133 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:18.890421 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:18.890427 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:18.890432 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:18.890641 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:18.891165 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:29:18.891170 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:29:18.891479 (+ 309us) client_negotiation.cc:769] Sending connection context
0923 17:29:18.891674 (+ 195us) client_negotiation.cc:240] Negotiation successful
0923 17:29:18.892015 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":151,"threads_started":1}
I20260923 17:29:18.894258 2535 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41487 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:29:18 dist-test-slave-ptl9 krb5kdc[2379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184558, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.61@KRBTEST.COM for kudu/127.11.229.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.229.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.60.keytab.
Entry for principal kudu/127.11.229.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.60.keytab.
I20260923 17:29:18.904058 2538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:18.894766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:41487 (local address 127.0.0.1:59360)
0923 17:29:18.894895 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:18.894910 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:18.894988 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:18.895618 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:18.895620 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:18.895797 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:18.895876 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:18.895883 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:18.896971 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:18.896977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:18.897652 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:18.897662 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:18.898372 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:18.898375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:18.898477 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:18.898955 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:18.898966 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:18.900213 (+ 1247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:29:18.902341 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:18.902344 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:18.902345 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:18.902609 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:18.902981 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:18.902989 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:18.902990 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:18.903078 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:18.903651 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:29:18.903653 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:29:18.903736 (+ 83us) client_negotiation.cc:769] Sending connection context
0923 17:29:18.903831 (+ 95us) client_negotiation.cc:240] Negotiation successful
0923 17:29:18.903932 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260923 17:29:18.904165 12180 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1jLtF9/build/debug/bin/kudu
/tmp/dist-test-task1jLtF9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.60:35841
--webserver_interface=127.11.229.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.60.keytab
--principal=kudu/127.11.229.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.229.20:44703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.229.62:34985,127.11.229.61:41487,127.11.229.60:35841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:29:18.904588 2540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:18.894936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59360 (local address 127.11.229.61:41487)
0923 17:29:18.895278 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:18.895289 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:18.895313 (+ 24us) server_negotiation.cc:407] Connection header received
0923 17:29:18.895346 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:18.895350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:18.895501 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:18.895641 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:18.895986 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:18.896853 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:18.897768 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:18.898261 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:18.898377 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:18.900353 (+ 1976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:18.900362 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:18.900373 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:18.900395 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:18.902208 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:18.902725 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:18.902730 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:18.902732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:18.902823 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:18.903193 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:18.903200 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:18.903203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:18.903525 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:18.903679 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:18.903971 (+ 292us) server_negotiation.cc:299] Negotiation successful
0923 17:29:18.904129 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":169,"threads_started":1}
I20260923 17:29:18.906569 2535 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.229.60" port: 35841 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:29:18.907379 2538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:18.907044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:35841 (local address 127.0.0.1:42592)
0923 17:29:18.907200 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:18.907293 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.60:35841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260923 17:29:18.907594 2483 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.229.60:35841: connect: Connection refused (error 111)
I20260923 17:29:18.908515 2538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:18.908217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:35841 (local address 127.0.0.1:42606)
0923 17:29:18.908359 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:18.908422 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.60:35841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260923 17:29:18.908711 2535 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.229.60:35841: Network error: Client connection negotiation failed: client connection to 127.11.229.60:35841: connect: Connection refused (error 111)
I20260923 17:29:18.958019 2535 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.229.60" port: 35841 } attempt: 1
I20260923 17:29:18.959014 2538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:18.958638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:35841 (local address 127.0.0.1:42616)
0923 17:29:18.958818 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:18.958913 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.60:35841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260923 17:29:18.959792 2538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:18.959553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:35841 (local address 127.0.0.1:42622)
0923 17:29:18.959673 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:18.959726 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.60:35841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 17:29:18.959944 2535 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.229.60:35841: Network error: Client connection negotiation failed: client connection to 127.11.229.60:35841: connect: Connection refused (error 111)
I20260923 17:29:18.996281 2460 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41487 } attempt: 4
Sep 23 17:29:18 dist-test-slave-ptl9 krb5kdc[2379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184558, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for kudu/127.11.229.61@KRBTEST.COM
I20260923 17:29:19.006774 2462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:18.996828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:41487 (local address 127.0.0.1:59366)
0923 17:29:18.997078 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:18.997098 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:18.997222 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:18.997563 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:18.997566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:18.997781 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:18.997918 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:18.997926 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:18.999214 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:18.999218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:19.000581 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:19.000595 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.001141 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.001145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:19.001286 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:19.001852 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:19.001868 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:19.003343 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:29:19.005129 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:19.005134 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:19.005136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:19.005433 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:19.005729 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:19.005732 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:19.005734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:19.005823 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:19.006221 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:29:19.006224 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:29:19.006338 (+ 114us) client_negotiation.cc:769] Sending connection context
0923 17:29:19.006447 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 17:29:19.006594 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142}
I20260923 17:29:19.006774 2540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:18.996979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59366 (local address 127.11.229.61:41487)
0923 17:29:18.997117 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:18.997121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:18.997158 (+ 37us) server_negotiation.cc:407] Connection header received
0923 17:29:18.997364 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:18.997367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:18.997425 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:18.997498 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:18.998039 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:18.999083 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.000730 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.001007 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.001099 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:19.003504 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:19.003508 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:19.003510 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:19.003536 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:19.004999 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:19.005533 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:19.005537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:19.005539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:19.005625 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:19.005927 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:19.005931 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:19.005933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:19.006121 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:19.006252 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:19.006514 (+ 262us) server_negotiation.cc:299] Negotiation successful
0923 17:29:19.006636 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 17:29:19.007716 2460 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.229.60" port: 35841 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:29:19.008477 2462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:19.008248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:35841 (local address 127.0.0.1:42634)
0923 17:29:19.008368 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:19.008417 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.60:35841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260923 17:29:19.009114 2462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:19.008907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:35841 (local address 127.0.0.1:42646)
0923 17:29:19.009012 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:19.009056 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.60:35841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260923 17:29:19.009234 2460 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.229.60:35841: Network error: Client connection negotiation failed: client connection to 127.11.229.60:35841: connect: Connection refused (error 111)
W20260923 17:29:19.021101 2541 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 17:29:19.021458 2541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:19.021533 2541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:19.021603 2541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:29:19.025652 2541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:29:19.025727 2541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:19.025772 2541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:19.025806 2541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:29:19.025851 2541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:29:19.028138 2535 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.229.60" port: 35841 } attempt: 2
I20260923 17:29:19.028913 2538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:19.028588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:35841 (local address 127.0.0.1:42660)
0923 17:29:19.028750 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:19.028820 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.60:35841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260923 17:29:19.029685 2538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:19.029455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:35841 (local address 127.0.0.1:42666)
0923 17:29:19.029570 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:19.029621 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.60:35841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260923 17:29:19.029825 2535 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.229.60:35841: Network error: Client connection negotiation failed: client connection to 127.11.229.60:35841: connect: Connection refused (error 111)
I20260923 17:29:19.031574 2541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.229.20:44703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.229.62:34985,127.11.229.61:41487,127.11.229.60:35841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/krb5kdc/kudu_127.11.229.60.keytab
--principal=kudu/127.11.229.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.229.60:35841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.229.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:29:19.032629 2541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:19.033782 2541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:19.042150 2548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:19.042140 2550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:19.042183 2547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:19.042577 2541 server_base.cc:1034] running on GCE node
I20260923 17:29:19.043066 2541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:19.044171 2541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:19.045339 2541 hybrid_clock.cc:648] HybridClock initialized: now 1790184559045319 us; error 31 us; skew 500 ppm
Sep 23 17:29:19 dist-test-slave-ptl9 krb5kdc[2379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184559, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:29:19.047946 2541 init.cc:377] Logged in from keytab as kudu/127.11.229.60@KRBTEST.COM (short username kudu)
I20260923 17:29:19.048775 2541 webserver.cc:460] Webserver started at http://127.11.229.60:34607/ using document root <none> and password file <none>
I20260923 17:29:19.049360 2541 fs_manager.cc:359] Metadata directory not provided
I20260923 17:29:19.049429 2541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:29:19.049600 2541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:29:19.051580 2541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-2/data/instance:
uuid: "794a87a507c24d918bf0ce3b62a5cb1e"
format_stamp: "Formatted at 2026-09-23 17:29:19 on dist-test-slave-ptl9"
server_key: "da6f309d6e256b245f951c7c66b65e1e"
server_key_iv: "6ded842481af9caa132da6fb00b7606b"
server_key_version: "encryptionkey@0"
I20260923 17:29:19.052078 2541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-2/wal/instance:
uuid: "794a87a507c24d918bf0ce3b62a5cb1e"
format_stamp: "Formatted at 2026-09-23 17:29:19 on dist-test-slave-ptl9"
server_key: "da6f309d6e256b245f951c7c66b65e1e"
server_key_iv: "6ded842481af9caa132da6fb00b7606b"
server_key_version: "encryptionkey@0"
I20260923 17:29:19.055758 2541 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 17:29:19.058163 2557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:19.059358 2541 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:29:19.059477 2541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-2/data,/tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-2/wal
uuid: "794a87a507c24d918bf0ce3b62a5cb1e"
format_stamp: "Formatted at 2026-09-23 17:29:19 on dist-test-slave-ptl9"
server_key: "da6f309d6e256b245f951c7c66b65e1e"
server_key_iv: "6ded842481af9caa132da6fb00b7606b"
server_key_version: "encryptionkey@0"
I20260923 17:29:19.059562 2541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:29:19.068411 2460 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.229.60" port: 35841 } attempt: 1
I20260923 17:29:19.069358 2462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:19.068945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:35841 (local address 127.0.0.1:42672)
0923 17:29:19.069132 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:19.069201 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.60:35841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260923 17:29:19.070168 2462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:19.069971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:35841 (local address 127.0.0.1:42674)
0923 17:29:19.070075 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:19.070120 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.229.60:35841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260923 17:29:19.070369 2460 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.229.60:35841: Network error: Client connection negotiation failed: client connection to 127.11.229.60:35841: connect: Connection refused (error 111)
I20260923 17:29:19.083696 2541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:29:19.087158 2541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:29:19.087319 2541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:29:19.094553 2541 rpc_server.cc:307] RPC server started. Bound to: 127.11.229.60:35841
I20260923 17:29:19.094844 2609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.229.60:35841 every 8 connection(s)
I20260923 17:29:19.095623 2541 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-2/data/info.pb
I20260923 17:29:19.098562 2610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:29:19.101184 12180 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1jLtF9/build/debug/bin/kudu as pid 2541
I20260923 17:29:19.101351 12180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1jLtF9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184271597484-12180-0/minicluster-data/master-2/wal/instance
I20260923 17:29:19.101711 12180 external_mini_cluster.cc:1428] Setting key f0451ab7440f410e75bf36564c9c7434
I20260923 17:29:19.102677 2610 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34985 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:29:19 dist-test-slave-ptl9 krb5kdc[2379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184559, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.60@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
Sep 23 17:29:19 dist-test-slave-ptl9 krb5kdc[2379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.60@KRBTEST.COM
I20260923 17:29:19.114859 2463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:19.104587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50852 (local address 127.11.229.62:34985)
0923 17:29:19.104724 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:19.104729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:19.105333 (+ 604us) server_negotiation.cc:407] Connection header received
0923 17:29:19.105394 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:19.105397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:19.105458 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:19.105566 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:19.106420 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.106916 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.107962 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.108222 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.108300 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:19.111030 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:19.111034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:19.111037 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:19.111065 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:19.112763 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:19.113304 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:19.113310 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:19.113312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:19.113398 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:19.113759 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:19.113764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:19.113765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:19.113964 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:19.114084 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:19.114555 (+ 471us) server_negotiation.cc:299] Negotiation successful
0923 17:29:19.114681 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 17:29:19.115669 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:19.103071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42686 (local address 127.11.229.60:35841)
0923 17:29:19.103542 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:19.103558 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:19.103594 (+ 36us) server_negotiation.cc:407] Connection header received
0923 17:29:19.104312 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:19.104338 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:19.104781 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:19.105089 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:19.105982 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.107165 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.108043 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.108520 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.108636 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:19.111512 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:19.111517 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:19.111527 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:19.111548 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:19.113046 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:19.113591 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:19.113596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:19.113597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:19.113669 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:19.113965 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:19.113970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:19.113977 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:19.114319 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:19.114526 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:19.114753 (+ 227us) server_negotiation.cc:299] Negotiation successful
0923 17:29:19.114898 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":330,"thread_start_us":144,"threads_started":1}
I20260923 17:29:19.115666 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:19.104506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.62:34985 (local address 127.0.0.1:50852)
0923 17:29:19.104878 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:19.104916 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:19.105069 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:19.105620 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:19.105624 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:19.105982 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:19.106273 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:19.106287 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.107025 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.107029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:19.107837 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:19.107845 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.108364 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.108368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:19.108588 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:19.109413 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:19.109430 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:19.110860 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:29:19.112899 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:19.112908 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:19.112918 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:19.113166 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:19.113510 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:19.113513 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:19.113515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:19.113653 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:19.114086 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:29:19.114096 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:29:19.114335 (+ 239us) client_negotiation.cc:769] Sending connection context
0923 17:29:19.114516 (+ 181us) client_negotiation.cc:240] Negotiation successful
0923 17:29:19.114820 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":140,"spinlock_wait_cycles":896,"thread_start_us":126,"threads_started":1}
I20260923 17:29:19.116943 2610 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41487 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:29:19.121021 2535 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.229.60" port: 35841 } attempt: 3
Sep 23 17:29:19 dist-test-slave-ptl9 krb5kdc[2379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184559, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.60@KRBTEST.COM for kudu/127.11.229.61@KRBTEST.COM
Sep 23 17:29:19 dist-test-slave-ptl9 krb5kdc[2379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.62@KRBTEST.COM
Sep 23 17:29:19 dist-test-slave-ptl9 krb5kdc[2379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184558, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.61@KRBTEST.COM for kudu/127.11.229.60@KRBTEST.COM
I20260923 17:29:19.126914 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:19.117479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.61:41487 (local address 127.0.0.1:59372)
0923 17:29:19.117571 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:19.117582 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:19.117678 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:19.117912 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:19.117914 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:19.118075 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:19.118210 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:19.118217 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.119009 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.119011 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:19.119536 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:19.119542 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.120095 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.120099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:19.120198 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:19.120671 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:19.120684 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:19.122245 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:29:19.124810 (+ 2565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:19.124815 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:19.124818 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:19.125433 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:19.125763 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:19.125766 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:19.125768 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:19.125868 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:19.126332 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:29:19.126336 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:29:19.126452 (+ 116us) client_negotiation.cc:769] Sending connection context
0923 17:29:19.126559 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 17:29:19.126691 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260923 17:29:19.126945 2540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:19.117554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59372 (local address 127.11.229.61:41487)
0923 17:29:19.117693 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:19.117700 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:19.117716 (+ 16us) server_negotiation.cc:407] Connection header received
0923 17:29:19.117761 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:19.117764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:19.117805 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:19.117885 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:19.118351 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.118872 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.119685 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.119977 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.120084 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:19.122389 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:19.122392 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:19.122395 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:19.122421 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:19.124697 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:19.125551 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:19.125555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:19.125557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:19.125647 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:19.125976 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:19.125980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:19.125981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:19.126217 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:19.126324 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:19.126640 (+ 316us) server_negotiation.cc:299] Negotiation successful
0923 17:29:19.126767 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 17:29:19.128095 2610 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.229.60" port: 35841 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:29:19.128201 2463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:19.118358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50866 (local address 127.11.229.62:34985)
0923 17:29:19.118475 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:19.118479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:19.118491 (+ 12us) server_negotiation.cc:407] Connection header received
0923 17:29:19.118585 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:19.118589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:19.118633 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:19.118714 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:19.119406 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.119958 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.120882 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.121099 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.121210 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:19.124320 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:19.124324 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:19.124327 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:19.124355 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:19.125965 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:19.126409 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:19.126413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:19.126415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:19.126505 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:19.126838 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:19.126842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:19.126844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:19.127074 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:19.127278 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:19.127908 (+ 630us) server_negotiation.cc:299] Negotiation successful
0923 17:29:19.128054 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 17:29:19.130191 2538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:19.121482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:35841 (local address 127.0.0.1:42698)
0923 17:29:19.121652 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:19.121664 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:19.121752 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:19.122024 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:19.122027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:19.122199 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:19.122272 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:19.122277 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.123292 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.123295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:19.124008 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:19.124019 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.124500 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.124503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:19.124597 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:19.125026 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:19.125042 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:19.126451 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:29:19.128488 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:19.128491 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:19.128493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:19.128763 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:19.129074 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:19.129077 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:19.129079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:19.129174 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:19.129659 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:29:19.129662 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:29:19.129752 (+ 90us) client_negotiation.cc:769] Sending connection context
0923 17:29:19.129857 (+ 105us) client_negotiation.cc:240] Negotiation successful
0923 17:29:19.130011 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260923 17:29:19.130213 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:19.121520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42698 (local address 127.11.229.60:35841)
0923 17:29:19.121619 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:19.121623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:19.121730 (+ 107us) server_negotiation.cc:407] Connection header received
0923 17:29:19.121825 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:19.121828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:19.121872 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:19.122001 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:19.122415 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.123169 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.124137 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.124387 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.124497 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:19.126597 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:19.126600 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:19.126603 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:19.126622 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:19.128358 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:19.128869 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:19.128874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:19.128878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:19.128960 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:19.129320 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:19.129323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:19.129325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:19.129528 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:19.129613 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:19.129913 (+ 300us) server_negotiation.cc:299] Negotiation successful
0923 17:29:19.130046 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 17:29:19.133948 2535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e254b9e5625e4bb7b8202b366208cb94: Bootstrap starting.
Sep 23 17:29:19 dist-test-slave-ptl9 krb5kdc[2379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.229.61@KRBTEST.COM
Sep 23 17:29:19 dist-test-slave-ptl9 krb5kdc[2379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184559, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.60@KRBTEST.COM for kudu/127.11.229.60@KRBTEST.COM
I20260923 17:29:19.137180 2535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e254b9e5625e4bb7b8202b366208cb94: Neither blocks nor log segments found. Creating new log.
I20260923 17:29:19.137562 2460 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.229.60" port: 35841 } attempt: 2
I20260923 17:29:19.138177 2535 log.cc:824] T 00000000000000000000000000000000 P e254b9e5625e4bb7b8202b366208cb94: Log is configured to *not* fsync() on all Append() calls
I20260923 17:29:19.138408 2540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:19.129681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59388 (local address 127.11.229.61:41487)
0923 17:29:19.129790 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:19.129794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:19.129805 (+ 11us) server_negotiation.cc:407] Connection header received
0923 17:29:19.129837 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:19.129840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:19.129882 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:19.129950 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:19.130770 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.131236 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.131991 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.132217 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.132320 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:19.134809 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:19.134814 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:19.134817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:19.134842 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:19.136123 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:19.136644 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:19.136648 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:19.136649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:19.136736 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:19.137028 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:19.137031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:19.137033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:19.137215 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:19.137487 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:19.138146 (+ 659us) server_negotiation.cc:299] Negotiation successful
0923 17:29:19.138259 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 17:29:19.140002 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:19.128540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:35841 (local address 127.0.0.1:42704)
0923 17:29:19.128691 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:19.128705 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:19.128775 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:19.129528 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:19.129531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:19.129690 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:19.129766 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:19.129773 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.131518 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.131521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:19.132214 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:19.132224 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.133139 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.133142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:19.133212 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:19.133665 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:19.133678 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:19.134958 (+ 1280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:29:19.137460 (+ 2502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:19.137463 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:19.137465 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:19.137791 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:19.138178 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:19.138181 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:19.138183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:19.138340 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:19.139558 (+ 1218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:29:19.139560 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:29:19.139649 (+ 89us) client_negotiation.cc:769] Sending connection context
0923 17:29:19.139749 (+ 100us) client_negotiation.cc:240] Negotiation successful
0923 17:29:19.139866 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260923 17:29:19.140486 2535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e254b9e5625e4bb7b8202b366208cb94: No bootstrap required, opened a new log
I20260923 17:29:19.141237 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:19.128765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42704 (local address 127.11.229.60:35841)
0923 17:29:19.129204 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:19.129210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:19.129219 (+ 9us) server_negotiation.cc:407] Connection header received
0923 17:29:19.129303 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:19.129306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:19.129348 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:19.129444 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:19.129882 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.131393 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.132335 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.132949 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.133044 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:19.135342 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:19.135346 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:19.135350 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:19.135378 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:19.137340 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:19.137885 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:19.137890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:19.137895 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:19.137973 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:19.138443 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:19.138447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:19.139129 (+ 682us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:19.139401 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:19.139517 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:19.140319 (+ 802us) server_negotiation.cc:299] Negotiation successful
0923 17:29:19.140438 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":103,"threads_started":1}
I20260923 17:29:19.142170 12180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Sep 23 17:29:19 dist-test-slave-ptl9 krb5kdc[2379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184558, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.229.62@KRBTEST.COM for kudu/127.11.229.60@KRBTEST.COM
I20260923 17:29:19.144752 2610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 794a87a507c24d918bf0ce3b62a5cb1e: Bootstrap starting.
I20260923 17:29:19.145203 2535 raft_consensus.cc:348] T 00000000000000000000000000000000 P e254b9e5625e4bb7b8202b366208cb94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3655778989524389bbb940717abeb9e3" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34985 } } peers { permanent_uuid: "e254b9e5625e4bb7b8202b366208cb94" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41487 } } peers { permanent_uuid: "794a87a507c24d918bf0ce3b62a5cb1e" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 35841 } }
I20260923 17:29:19.145505 2535 raft_consensus.cc:729] T 00000000000000000000000000000000 P e254b9e5625e4bb7b8202b366208cb94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e254b9e5625e4bb7b8202b366208cb94, State: Initialized, Role: FOLLOWER
I20260923 17:29:19.145859 2535 consensus_queue.cc:260] T 00000000000000000000000000000000 P e254b9e5625e4bb7b8202b366208cb94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3655778989524389bbb940717abeb9e3" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34985 } } peers { permanent_uuid: "e254b9e5625e4bb7b8202b366208cb94" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41487 } } peers { permanent_uuid: "794a87a507c24d918bf0ce3b62a5cb1e" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 35841 } }
I20260923 17:29:19.146718 2462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:19.137934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.229.60:35841 (local address 127.0.0.1:42710)
0923 17:29:19.138113 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:19.138127 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:19.138204 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:19.138707 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:19.138710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:19.138879 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:19.138967 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:19.138973 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.139901 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.139905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:19.140651 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:19.140664 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.141306 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.141310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:19.141390 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:19.141771 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:19.141785 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:19.143175 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:29:19.145055 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:19.145059 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:19.145061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:19.145367 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:19.145692 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:19.145695 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:19.145697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:19.145783 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:19.146234 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:29:19.146237 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:29:19.146331 (+ 94us) client_negotiation.cc:769] Sending connection context
0923 17:29:19.146440 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 17:29:19.146573 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260923 17:29:19.145479 2434 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3655778989524389bbb940717abeb9e3"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50866
I20260923 17:29:19.147200 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:19.138392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42710 (local address 127.11.229.60:35841)
0923 17:29:19.138499 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:19.138503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:19.138516 (+ 13us) server_negotiation.cc:407] Connection header received
0923 17:29:19.138555 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:19.138560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:19.138606 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:19.138683 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:19.139077 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.139769 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.140791 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.141083 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.141174 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:19.143321 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:19.143325 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:19.143328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:19.143353 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:19.144931 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:19.145495 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:19.145498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:19.145501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:19.145587 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:19.145883 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:19.145887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:19.145889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:19.146085 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:19.146186 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:19.146964 (+ 778us) server_negotiation.cc:299] Negotiation successful
0923 17:29:19.147072 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 17:29:19.147318 2535 sys_catalog.cc:564] T 00000000000000000000000000000000 P e254b9e5625e4bb7b8202b366208cb94 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:29:19.147526 2610 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 794a87a507c24d918bf0ce3b62a5cb1e: Neither blocks nor log segments found. Creating new log.
I20260923 17:29:19.147727 2618 sys_catalog.cc:455] T 00000000000000000000000000000000 P e254b9e5625e4bb7b8202b366208cb94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3655778989524389bbb940717abeb9e3" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34985 } } peers { permanent_uuid: "e254b9e5625e4bb7b8202b366208cb94" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41487 } } peers { permanent_uuid: "794a87a507c24d918bf0ce3b62a5cb1e" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 35841 } } }
I20260923 17:29:19.147848 2618 sys_catalog.cc:458] T 00000000000000000000000000000000 P e254b9e5625e4bb7b8202b366208cb94 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:29:19.148437 2610 log.cc:824] T 00000000000000000000000000000000 P 794a87a507c24d918bf0ce3b62a5cb1e: Log is configured to *not* fsync() on all Append() calls
I20260923 17:29:19.150249 2460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3655778989524389bbb940717abeb9e3: Bootstrap starting.
I20260923 17:29:19.150686 2610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 794a87a507c24d918bf0ce3b62a5cb1e: No bootstrap required, opened a new log
I20260923 17:29:19.152943 2460 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3655778989524389bbb940717abeb9e3: Neither blocks nor log segments found. Creating new log.
W20260923 17:29:19.153064 2630 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e254b9e5625e4bb7b8202b366208cb94: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:29:19.153129 2630 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:29:19.153874 2460 log.cc:824] T 00000000000000000000000000000000 P 3655778989524389bbb940717abeb9e3: Log is configured to *not* fsync() on all Append() calls
I20260923 17:29:19.151149 2467 master_runner.cc:186] Error getting master registration for 127.11.229.62:34985: OK, instance_id { permanent_uuid: "3655778989524389bbb940717abeb9e3" instance_seqno: 1790184558607277 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 17:29:19.154495 2467 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 17:29:19.154798 2610 raft_consensus.cc:348] T 00000000000000000000000000000000 P 794a87a507c24d918bf0ce3b62a5cb1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3655778989524389bbb940717abeb9e3" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34985 } } peers { permanent_uuid: "e254b9e5625e4bb7b8202b366208cb94" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41487 } } peers { permanent_uuid: "794a87a507c24d918bf0ce3b62a5cb1e" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 35841 } }
I20260923 17:29:19.154989 2610 raft_consensus.cc:729] T 00000000000000000000000000000000 P 794a87a507c24d918bf0ce3b62a5cb1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 794a87a507c24d918bf0ce3b62a5cb1e, State: Initialized, Role: FOLLOWER
I20260923 17:29:19.155237 2610 consensus_queue.cc:260] T 00000000000000000000000000000000 P 794a87a507c24d918bf0ce3b62a5cb1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3655778989524389bbb940717abeb9e3" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34985 } } peers { permanent_uuid: "e254b9e5625e4bb7b8202b366208cb94" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41487 } } peers { permanent_uuid: "794a87a507c24d918bf0ce3b62a5cb1e" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 35841 } }
I20260923 17:29:19.155642 2633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 794a87a507c24d918bf0ce3b62a5cb1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3655778989524389bbb940717abeb9e3" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34985 } } peers { permanent_uuid: "e254b9e5625e4bb7b8202b366208cb94" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41487 } } peers { permanent_uuid: "794a87a507c24d918bf0ce3b62a5cb1e" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 35841 } } }
I20260923 17:29:19.155778 2633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 794a87a507c24d918bf0ce3b62a5cb1e [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:29:19.156399 2610 sys_catalog.cc:564] T 00000000000000000000000000000000 P 794a87a507c24d918bf0ce3b62a5cb1e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:29:19.159205 2460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3655778989524389bbb940717abeb9e3: No bootstrap required, opened a new log
I20260923 17:29:19.162364 2460 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3655778989524389bbb940717abeb9e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3655778989524389bbb940717abeb9e3" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34985 } } peers { permanent_uuid: "e254b9e5625e4bb7b8202b366208cb94" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41487 } } peers { permanent_uuid: "794a87a507c24d918bf0ce3b62a5cb1e" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 35841 } }
I20260923 17:29:19.162595 2460 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3655778989524389bbb940717abeb9e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3655778989524389bbb940717abeb9e3, State: Initialized, Role: FOLLOWER
I20260923 17:29:19.159895 2541 master_runner.cc:186] Error getting master registration for 127.11.229.62:34985: OK, instance_id { permanent_uuid: "3655778989524389bbb940717abeb9e3" instance_seqno: 1790184558607277 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 17:29:19.163291 2541 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 17:29:19.163290 2460 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3655778989524389bbb940717abeb9e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3655778989524389bbb940717abeb9e3" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34985 } } peers { permanent_uuid: "e254b9e5625e4bb7b8202b366208cb94" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41487 } } peers { permanent_uuid: "794a87a507c24d918bf0ce3b62a5cb1e" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 35841 } }
W20260923 17:29:19.163676 2644 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 794a87a507c24d918bf0ce3b62a5cb1e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:29:19.163758 2644 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:29:19.164112 2646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3655778989524389bbb940717abeb9e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3655778989524389bbb940717abeb9e3" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34985 } } peers { permanent_uuid: "e254b9e5625e4bb7b8202b366208cb94" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41487 } } peers { permanent_uuid: "794a87a507c24d918bf0ce3b62a5cb1e" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 35841 } } }
I20260923 17:29:19.164251 2646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3655778989524389bbb940717abeb9e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:29:19.165128 2460 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3655778989524389bbb940717abeb9e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:29:19.165161 2434 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3655778989524389bbb940717abeb9e3 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 17:29:19.165341 2434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3655778989524389bbb940717abeb9e3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:29:19.166628 2434 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3655778989524389bbb940717abeb9e3 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3655778989524389bbb940717abeb9e3" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34985 } } peers { permanent_uuid: "e254b9e5625e4bb7b8202b366208cb94" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41487 } } peers { permanent_uuid: "794a87a507c24d918bf0ce3b62a5cb1e" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 35841 } }
I20260923 17:29:19.167642 2434 leader_election.cc:288] T 00000000000000000000000000000000 P 3655778989524389bbb940717abeb9e3 [CANDIDATE]: Term 1 election: Requested vote from peers e254b9e5625e4bb7b8202b366208cb94 (127.11.229.61:41487), 794a87a507c24d918bf0ce3b62a5cb1e (127.11.229.60:35841)
I20260923 17:29:19.168219 2509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3655778989524389bbb940717abeb9e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e254b9e5625e4bb7b8202b366208cb94"
I20260923 17:29:19.168355 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3655778989524389bbb940717abeb9e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "794a87a507c24d918bf0ce3b62a5cb1e"
I20260923 17:29:19.168512 2509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e254b9e5625e4bb7b8202b366208cb94 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:29:19.168617 2584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 794a87a507c24d918bf0ce3b62a5cb1e [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:29:19.169932 2509 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e254b9e5625e4bb7b8202b366208cb94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3655778989524389bbb940717abeb9e3 in term 1.
I20260923 17:29:19.170007 2584 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 794a87a507c24d918bf0ce3b62a5cb1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3655778989524389bbb940717abeb9e3 in term 1.
I20260923 17:29:19.170332 2410 leader_election.cc:302] T 00000000000000000000000000000000 P 3655778989524389bbb940717abeb9e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3655778989524389bbb940717abeb9e3, e254b9e5625e4bb7b8202b366208cb94; no voters:
I20260923 17:29:19.170909 2646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3655778989524389bbb940717abeb9e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:29:19.171147 2646 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3655778989524389bbb940717abeb9e3 [term 1 LEADER]: Becoming Leader. State: Replica: 3655778989524389bbb940717abeb9e3, State: Running, Role: LEADER
I20260923 17:29:19.171526 2646 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3655778989524389bbb940717abeb9e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3655778989524389bbb940717abeb9e3" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34985 } } peers { permanent_uuid: "e254b9e5625e4bb7b8202b366208cb94" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41487 } } peers { permanent_uuid: "794a87a507c24d918bf0ce3b62a5cb1e" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 35841 } }
W20260923 17:29:19.173326 2657 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3655778989524389bbb940717abeb9e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:29:19.173444 2657 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:29:19.174111 2646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3655778989524389bbb940717abeb9e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3655778989524389bbb940717abeb9e3. Latest consensus state: current_term: 1 leader_uuid: "3655778989524389bbb940717abeb9e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3655778989524389bbb940717abeb9e3" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34985 } } peers { permanent_uuid: "e254b9e5625e4bb7b8202b366208cb94" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41487 } } peers { permanent_uuid: "794a87a507c24d918bf0ce3b62a5cb1e" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 35841 } } }
I20260923 17:29:19.174216 2646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3655778989524389bbb940717abeb9e3 [sys.catalog]: This master's current role is: LEADER
I20260923 17:29:19.174822 2668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:29:19.175737 2668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:29:19.176061 2392 master_runner.cc:263] No leader master found from master 3655778989524389bbb940717abeb9e3
I20260923 17:29:19.176124 2392 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 17:29:19.182019 2540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:19.169952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59400 (local address 127.11.229.61:41487)
0923 17:29:19.170093 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:19.170097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:19.172114 (+ 2017us) server_negotiation.cc:407] Connection header received
0923 17:29:19.172310 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:19.172313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:19.172364 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:19.172419 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:19.173486 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.174208 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.176014 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.176330 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.176408 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:19.178345 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:19.178349 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:19.178352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:19.178377 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:19.180127 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:19.180642 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:19.180645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:19.180647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:19.180731 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:19.181026 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:19.181030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:19.181031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:19.181239 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:19.181364 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:19.181740 (+ 376us) server_negotiation.cc:299] Negotiation successful
0923 17:29:19.181866 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:29:19.182790 2584 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 794a87a507c24d918bf0ce3b62a5cb1e [term 1 FOLLOWER]: Refusing update from remote peer 3655778989524389bbb940717abeb9e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:29:19.184090 2509 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e254b9e5625e4bb7b8202b366208cb94 [term 1 FOLLOWER]: Refusing update from remote peer 3655778989524389bbb940717abeb9e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:29:19.184511 2646 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3655778989524389bbb940717abeb9e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "794a87a507c24d918bf0ce3b62a5cb1e" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:29:19.184958 2646 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3655778989524389bbb940717abeb9e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e254b9e5625e4bb7b8202b366208cb94" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:29:19.185199 2463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:19.170915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50868 (local address 127.11.229.62:34985)
0923 17:29:19.171040 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:19.171045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:19.173582 (+ 2537us) server_negotiation.cc:407] Connection header received
0923 17:29:19.173696 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:19.173699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:19.173737 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:19.173831 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:19.174703 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.175374 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.176468 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.176769 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.176841 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:19.178524 (+ 1683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:19.178528 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:19.178531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:19.178557 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:19.180276 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:19.180777 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:19.180781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:19.180783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:19.180866 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:19.181158 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:19.181161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:19.181163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:19.181409 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:19.181525 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:19.184910 (+ 3385us) server_negotiation.cc:299] Negotiation successful
0923 17:29:19.185048 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 17:29:19.189249 2618 sys_catalog.cc:455] T 00000000000000000000000000000000 P e254b9e5625e4bb7b8202b366208cb94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3655778989524389bbb940717abeb9e3. Latest consensus state: current_term: 1 leader_uuid: "3655778989524389bbb940717abeb9e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3655778989524389bbb940717abeb9e3" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34985 } } peers { permanent_uuid: "e254b9e5625e4bb7b8202b366208cb94" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41487 } } peers { permanent_uuid: "794a87a507c24d918bf0ce3b62a5cb1e" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 35841 } } }
I20260923 17:29:19.189404 2618 sys_catalog.cc:458] T 00000000000000000000000000000000 P e254b9e5625e4bb7b8202b366208cb94 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:29:19.192370 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:19.171528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42716 (local address 127.11.229.60:35841)
0923 17:29:19.171647 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:19.171651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:19.182311 (+ 10660us) server_negotiation.cc:407] Connection header received
0923 17:29:19.182358 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:19.182361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:19.182410 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:19.182480 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:19.183719 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.184476 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.186316 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.186636 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.186740 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:19.188389 (+ 1649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:19.188393 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:19.188396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:19.188436 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:19.190303 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:19.190814 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:19.190818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:19.190820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:19.190906 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:19.191197 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:19.191201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:19.191202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:19.191412 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:19.191540 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:19.192124 (+ 584us) server_negotiation.cc:299] Negotiation successful
0923 17:29:19.192231 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 17:29:19.192775 2633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 794a87a507c24d918bf0ce3b62a5cb1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3655778989524389bbb940717abeb9e3. Latest consensus state: current_term: 1 leader_uuid: "3655778989524389bbb940717abeb9e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3655778989524389bbb940717abeb9e3" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34985 } } peers { permanent_uuid: "e254b9e5625e4bb7b8202b366208cb94" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41487 } } peers { permanent_uuid: "794a87a507c24d918bf0ce3b62a5cb1e" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 35841 } } }
I20260923 17:29:19.192898 2633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 794a87a507c24d918bf0ce3b62a5cb1e [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:29:19.194218 2669 mvcc.cc:205] Tried to move back new op lower bound from 7332595954405920768 to 7332595954370211840. Current Snapshot: MvccSnapshot[applied={T|T < 7332595954405920768}]
I20260923 17:29:19.194413 2663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3655778989524389bbb940717abeb9e3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3655778989524389bbb940717abeb9e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3655778989524389bbb940717abeb9e3" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34985 } } peers { permanent_uuid: "e254b9e5625e4bb7b8202b366208cb94" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41487 } } peers { permanent_uuid: "794a87a507c24d918bf0ce3b62a5cb1e" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 35841 } } }
I20260923 17:29:19.194540 2663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3655778989524389bbb940717abeb9e3 [sys.catalog]: This master's current role is: LEADER
I20260923 17:29:19.197170 2646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3655778989524389bbb940717abeb9e3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3655778989524389bbb940717abeb9e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3655778989524389bbb940717abeb9e3" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34985 } } peers { permanent_uuid: "e254b9e5625e4bb7b8202b366208cb94" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41487 } } peers { permanent_uuid: "794a87a507c24d918bf0ce3b62a5cb1e" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 35841 } } }
I20260923 17:29:19.197319 2646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3655778989524389bbb940717abeb9e3 [sys.catalog]: This master's current role is: LEADER
I20260923 17:29:19.198252 2668 catalog_manager.cc:1347] Generated new cluster ID: 233ec0d6481c455db2297249363a8248
I20260923 17:29:19.198228 2673 mvcc.cc:205] Tried to move back new op lower bound from 7332595954405920768 to 7332595954370211840. Current Snapshot: MvccSnapshot[applied={T|T < 7332595954405920768}]
I20260923 17:29:19.198362 2668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:29:19.198377 2633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 794a87a507c24d918bf0ce3b62a5cb1e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3655778989524389bbb940717abeb9e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3655778989524389bbb940717abeb9e3" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34985 } } peers { permanent_uuid: "e254b9e5625e4bb7b8202b366208cb94" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41487 } } peers { permanent_uuid: "794a87a507c24d918bf0ce3b62a5cb1e" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 35841 } } }
I20260923 17:29:19.198475 2633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 794a87a507c24d918bf0ce3b62a5cb1e [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:29:19.203362 2618 sys_catalog.cc:455] T 00000000000000000000000000000000 P e254b9e5625e4bb7b8202b366208cb94 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3655778989524389bbb940717abeb9e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3655778989524389bbb940717abeb9e3" member_type: VOTER last_known_addr { host: "127.11.229.62" port: 34985 } } peers { permanent_uuid: "e254b9e5625e4bb7b8202b366208cb94" member_type: VOTER last_known_addr { host: "127.11.229.61" port: 41487 } } peers { permanent_uuid: "794a87a507c24d918bf0ce3b62a5cb1e" member_type: VOTER last_known_addr { host: "127.11.229.60" port: 35841 } } }
I20260923 17:29:19.203493 2618 sys_catalog.cc:458] T 00000000000000000000000000000000 P e254b9e5625e4bb7b8202b366208cb94 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:29:19.217322 2668 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:29:19.230571 2668 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:29:19.247928 2668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3655778989524389bbb940717abeb9e3: Generated new TSK 0
I20260923 17:29:19.249810 2668 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 17:29:19.279328 2540 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:19.276061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59412 (local address 127.11.229.61:41487)
0923 17:29:19.276195 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:19.276201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:19.277946 (+ 1745us) server_negotiation.cc:407] Connection header received
0923 17:29:19.278408 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:19.278413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:19.278468 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:19.278539 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:19.279184 (+ 645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59412: BlockingRecv error: recv got EOF from 127.0.0.1:59412 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260923 17:29:19.280781 2463 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:19.276928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50870 (local address 127.11.229.62:34985)
0923 17:29:19.277048 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:19.277053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:19.279834 (+ 2781us) server_negotiation.cc:407] Connection header received
0923 17:29:19.280002 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:19.280006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:19.280061 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:19.280125 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:19.280655 (+ 530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50870: BlockingRecv error: recv got EOF from 127.0.0.1:50870 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260923 17:29:19.282191 2613 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:19.277602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42718 (local address 127.11.229.60:35841)
0923 17:29:19.277714 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:19.277718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:19.281178 (+ 3460us) server_negotiation.cc:407] Connection header received
0923 17:29:19.281465 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:19.281469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:19.281531 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:19.281583 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:19.282069 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42718: BlockingRecv error: recv got EOF from 127.0.0.1:42718 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260923 17:29:19.286845 2540 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:19.285186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59428 (local address 127.11.229.61:41487)
0923 17:29:19.285354 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:19.285358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:19.285874 (+ 516us) server_negotiation.cc:407] Connection header received
0923 17:29:19.286099 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:19.286103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:19.286184 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:19.286235 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:19.286742 (+ 507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59428: BlockingRecv error: recv got EOF from 127.0.0.1:59428 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260923 17:29:19.292248 2540 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:19.290577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59432 (local address 127.11.229.61:41487)
0923 17:29:19.290683 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:19.290687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:19.291279 (+ 592us) server_negotiation.cc:407] Connection header received
0923 17:29:19.291465 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:19.291469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:19.291550 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:19.291602 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:19.292136 (+ 534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59432: BlockingRecv error: recv got EOF from 127.0.0.1:59432 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260923 17:29:19.297892 2540 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:19.296292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59438 (local address 127.11.229.61:41487)
0923 17:29:19.296401 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:19.296405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:19.296940 (+ 535us) server_negotiation.cc:407] Connection header received
0923 17:29:19.297133 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:19.297136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:19.297186 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:19.297236 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:19.297780 (+ 544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59438: BlockingRecv error: recv got EOF from 127.0.0.1:59438 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260923 17:29:19.305672 2540 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:19.303952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59442 (local address 127.11.229.61:41487)
0923 17:29:19.304110 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:19.304115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:19.304649 (+ 534us) server_negotiation.cc:407] Connection header received
0923 17:29:19.304838 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:19.304842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:19.304893 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:19.304945 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:19.305560 (+ 615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59442: BlockingRecv error: recv got EOF from 127.0.0.1:59442 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260923 17:29:19.317709 2540 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:19.315977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59448 (local address 127.11.229.61:41487)
0923 17:29:19.316099 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:19.316104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:19.316673 (+ 569us) server_negotiation.cc:407] Connection header received
0923 17:29:19.316876 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:19.316880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:19.316933 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:19.316984 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:19.317595 (+ 611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59448: BlockingRecv error: recv got EOF from 127.0.0.1:59448 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260923 17:29:19.338284 2540 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:19.336281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59462 (local address 127.11.229.61:41487)
0923 17:29:19.336417 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:19.336422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:19.337075 (+ 653us) server_negotiation.cc:407] Connection header received
0923 17:29:19.337350 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:19.337356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:19.337419 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:19.337513 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:19.338155 (+ 642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59462: BlockingRecv error: recv got EOF from 127.0.0.1:59462 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260923 17:29:19.375512 2540 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:19.373767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59472 (local address 127.11.229.61:41487)
0923 17:29:19.373898 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:19.373903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:19.374506 (+ 603us) server_negotiation.cc:407] Connection header received
0923 17:29:19.374722 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:19.374726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:19.374782 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:19.374837 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:19.375369 (+ 532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59472: BlockingRecv error: recv got EOF from 127.0.0.1:59472 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260923 17:29:19.443508 2540 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:19.442163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59488 (local address 127.11.229.61:41487)
0923 17:29:19.442329 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:19.442336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:19.442577 (+ 241us) server_negotiation.cc:407] Connection header received
0923 17:29:19.442762 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:19.442765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:19.442816 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:19.442911 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:19.443390 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59488: BlockingRecv error: recv got EOF from 127.0.0.1:59488 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260923 17:29:19.574940 2540 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:19.573727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59502 (local address 127.11.229.61:41487)
0923 17:29:19.573897 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:19.573902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:19.574037 (+ 135us) server_negotiation.cc:407] Connection header received
0923 17:29:19.574210 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:19.574214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:19.574271 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:19.574367 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:19.574802 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59502: BlockingRecv error: recv got EOF from 127.0.0.1:59502 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260923 17:29:19.834364 2540 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:19.833194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59514 (local address 127.11.229.61:41487)
0923 17:29:19.833429 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:19.833433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:19.833450 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:29:19.833591 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:19.833594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:19.833643 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:19.833716 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:19.834181 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59514: BlockingRecv error: recv got EOF from 127.0.0.1:59514 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
I20260923 17:29:20.154520 2630 catalog_manager.cc:1259] Loaded cluster ID: 233ec0d6481c455db2297249363a8248
I20260923 17:29:20.154636 2630 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e254b9e5625e4bb7b8202b366208cb94: loading cluster ID for follower catalog manager: success
I20260923 17:29:20.156352 2630 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e254b9e5625e4bb7b8202b366208cb94: acquiring CA information for follower catalog manager: success
I20260923 17:29:20.157644 2630 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e254b9e5625e4bb7b8202b366208cb94: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 17:29:20.164851 2644 catalog_manager.cc:1259] Loaded cluster ID: 233ec0d6481c455db2297249363a8248
I20260923 17:29:20.164913 2644 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 794a87a507c24d918bf0ce3b62a5cb1e: loading cluster ID for follower catalog manager: success
I20260923 17:29:20.166682 2644 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 794a87a507c24d918bf0ce3b62a5cb1e: acquiring CA information for follower catalog manager: success
I20260923 17:29:20.167843 2644 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 794a87a507c24d918bf0ce3b62a5cb1e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 17:29:20.352516 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:20.348419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59524 (local address 127.11.229.61:41487)
0923 17:29:20.348697 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:20.348704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:20.348807 (+ 103us) server_negotiation.cc:407] Connection header received
0923 17:29:20.348987 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:20.348991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:20.349133 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:20.349299 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:29:20.349595 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:20.350136 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:20.351000 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:20.351230 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:20.351339 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:20.351678 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:29:20.352119 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:29:20.352253 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:20.352295 (+ 42us) server_negotiation.cc:299] Negotiation successful
0923 17:29:20.352350 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":78,"threads_started":1}
I20260923 17:29:20.359182 2748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:20.355440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42720 (local address 127.11.229.60:35841)
0923 17:29:20.355702 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:20.355707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:20.355725 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:29:20.355808 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:20.355811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:20.355940 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:20.356032 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:29:20.356205 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:20.356761 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:20.357688 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:20.358000 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:20.358082 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:20.358382 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:29:20.358860 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:29:20.358951 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:20.359016 (+ 65us) server_negotiation.cc:299] Negotiation successful
0923 17:29:20.359066 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
I20260923 17:29:20.366623 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:20.362076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59528 (local address 127.11.229.61:41487)
0923 17:29:20.362206 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:20.362210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:20.362379 (+ 169us) server_negotiation.cc:407] Connection header received
0923 17:29:20.362604 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:20.362607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:20.362668 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:20.362722 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:29:20.363121 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:20.363902 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:20.365379 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:20.365711 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:20.365816 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:20.366122 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:29:20.366198 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:29:20.366276 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:20.366440 (+ 164us) server_negotiation.cc:299] Negotiation successful
0923 17:29:20.366498 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 17:29:20.366878 2748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:20.362746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42734 (local address 127.11.229.60:35841)
0923 17:29:20.362870 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:20.362874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:20.362980 (+ 106us) server_negotiation.cc:407] Connection header received
0923 17:29:20.363177 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:20.363181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:20.363226 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:20.363283 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:29:20.363635 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:20.364289 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:20.365378 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:20.365723 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:20.365809 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:20.366368 (+ 559us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:29:20.366445 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:29:20.366511 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:20.366634 (+ 123us) server_negotiation.cc:299] Negotiation successful
0923 17:29:20.366681 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 17:29:20.367779 2754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:20.362120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50886 (local address 127.11.229.62:34985)
0923 17:29:20.362376 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:20.362381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:20.362647 (+ 266us) server_negotiation.cc:407] Connection header received
0923 17:29:20.362741 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:20.362744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:20.362955 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:20.363081 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:29:20.363315 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:20.364039 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:20.365312 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:20.365705 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:20.365852 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:20.366445 (+ 593us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:29:20.367027 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:29:20.367104 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:20.367406 (+ 302us) server_negotiation.cc:299] Negotiation successful
0923 17:29:20.367676 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":65,"threads_started":1}
I20260923 17:29:20.370954 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 2392
I20260923 17:29:20.378070 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 2467
I20260923 17:29:20.384166 12180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1jLtF9/build/debug/bin/kudu with pid 2541
2026-09-23T17:29:20Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3123 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6251 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (288779 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST